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Summary 

— T>  Estimation  of  rmi  target  position  accuracy  for  a  radar  la  of  great  importance 
to  radar  manufacturers  and  customers  alike.  With  increasing  frequency  customers 
insist  on  direct  measurement  with  small  RCS  aircraft  on  radial  flight  paths.  Too 
often,  however,  a  colocated  precision  reference  radar  is  unavailable  from  which  to 
accurately  define  the  true  target  flight  path.  In  these  cases  the  total  error  can  be 
tested  in  two  parts:  the  bias  component  can  be  estimated  from  static  measurements 
and  the  random  component  {Jitter  and  thermal  error)  is  estimated  as  a  variance. 

[Here  the  true  target  flight  path  is  traditiorally  modelled  as  entirely  radial.  Vnfortun-* 
stei/  «.  .->*»  small  deviations  from  a  true  radial  can  lead  to  Urge  errors  in  variance 
estimation,  particularly  when  the  target  is  close  to  the  radar  where  the  radar  error  H 
expected  to  be  smalt.  If  the  customer  requires  proof  of  theoretical  sccuracy  then  the  ‘ 
njodel  error  can  be  larger  than  die  radar  error  and  the  radar  will  falsely  fsil  the  test 


The  approach  to  variance  estimation  developed  here  is  to  estimate  the  mean 
target  path  along  with  the  tri-coordinate  position  variance  and  thus  avoid  the  error 
associated  with  assuming  a  simple  flight  path  model.  The  method  is  quite  generajdro 
sensors  and  targets  of  all  types  even  though  the  specific  application  it  wasde^doped 
for  was  a  radar  under  test  with  an  aircraft  on  a  radial  path.  The  mgaw-pfiTh  it  as¬ 
sumed  to  be  a  su.n  of  orthogonal  polynomials  of  order  hTjLfw'Show  i  that  so  long  as 
M  is  greater  than  or  equal  to  K,  the  effective  true^whtf<o?the  mean  path  (overrating)] 
that  the  variance  estimate  is  unbiased  in  th^jirtTan.  Moreover  the  adjacent-point 
correlation  coefficient  is  shown  to  be  a  sensitive  indicator  of  overfittlng.  The  method 
is  shown  to  be  practical  in  the  face  of  simulation  and  real  flight  data  even  with  mi.»e- 
lng  data  points  as  from  missed  detections^iln  addition  to  unbiased  optimal  estima¬ 
tion  of  variance,  expressions  are  developed  for  the  uncertainty  in  the  estimate  and 
related  to  the  producer's  and  consumer's  ilsks  of  falsely  failing  or  falsely  passing  a 
tri -coordinate  position  accuracy  test.  Considerable  development  is  accorded  sound 
test  design  with  .hese  principles,  and  expressions  are  developed  for  confidence  limits 
to  infer  bounds  or  true  variance  given  the  test  results. 


>i  iC 

^ELECTE 


pTOrdocumenl  has 

l  iii'hlic  release  and  sale, 


♦or  public  release 

1  £* 


6 1 


C  'j 

1  O 


00  i 


*  j# 


GENERAL  ELECTRIC  COMPANY  TECHNICAL  INFORMATION 


Within  the  limitations  imposed  by  Government  data  export  regulations  and  security 
classifications,  the  availability  of  General  Electric  Company  technical  information 
is  regulated  by  the  following  classifications  In  order  to  safeguard  proprietary  infor¬ 
mation: 


CLASS  1:  GENERAL  INFORMATION 

Available  to  anyone  on  request  - 
Patent,  legal  and  commercial  review 
required  before  issue. 

CLASS  2:  GENERAL  COMPANY  INFORMATION 

Available  to  any  General  Electric  Company 
employee  on  request. 

Available  to  any  General  Electric  Subsidiary 
or  Licensee  subject  to  existing  agreements. 

Disclosure  outside  General  Electric  Company 
requires  approval  of  originating  component. 

CLASS  3:  LIMITED  AVAILABILITY  INFORMATION 

Original  Distribution  to  those  individuals  with 
specific  need  for  Information. 

Subsequent  Company  availability  requires 
originating  component  approval . 

Disclosure  outside  General  Electric  Company 
-  requires  approval  of  originating  component . 

CLASS  4:  HIGHLY  RESTRICTED  DISTRIBUTION 

Original  distribution  to  those  individuals  person¬ 
ally  responsible  for  the  Company's  interests  in 
the  subject. 

Copies  serially  numbered,  assigned  and  recorded 
by  name. 

Material  content,  and  knowledge  of  existence, 
restricted  to  copy  holder . 

GOVERNMENT  SECURITY  CLASSIFICATIONS,  when  required,  take  precedence 
in  the  handling  of  the  material.  Wherever  not  specifically  disallow  3d.  the  General 
Electric  classifications  should  also  be  included  in  order  to  obtain  proper  handling 
routines. 


MME 0  40238  (5-65) 


GENERAL  ELECTRIC  COMPANY 
MILITARY  ELECTRONIC  EY8TEMS  OPERATIONS 
TECHNICAL  INFORMATION  SERIES 


SECTION 
UNIT _ _ 


Engineering  Operations 


MESC  ACCOUNTING  REFERENCE 


Radar  Svstofpq  Fttyineering 
510 


COLLABORATORS 

APPROVED 


Dr.  B.A.  Deresh  and  R.  J.  Anderson 


tle  Mgr..  RSE  location  CSP  5-C4,  Syracuse,  NY 


R.Xi.  Benfey 


MINIMUM  DISTRIBUTION  •  C»»rn»*.il  'Jnc!r-**liad  Mtti fiol  (and  Ti*la  Pofl**)  in  G.E.  Cla*»a»  I,  2,  oi  3  will  ba  the 

following, 


Tltln  Pe«i  Only 

- T - 

1 

6 


To 

Logoi  Section,  W£«0  (SyracuM) 

V.*nag*r,  Toehnologlial  Planning,  MESO  (Syracu*a) 
G-E  Technical  Data  Cantor  (Schanactcdyi 


MINIMUM  DISTRIBUTION  •  Govotnmont  Clottifiod  Motgrial,  Sat rat  or  Con fi n'en ti ol  in  G.E.  Clattat  I,  2,  or  3  will  ba 

Ik*  lollowing. 


1  0  Managor,  Tachnologlcal  Planning,  MESO  tLv'OuuM) 

ADDIyIONAL  DISTRIBUTION  (Kaap  of  minimum  within  infant  ol  ottigned  C.c.  Clots.) 


COPIES 

NAME 

LOCATION 

5 

DEFENSE  DOCUMENTATION 

CAMERON  STATION, 

(CLASS  1  ONLY) 

CENTER 

ALEXANDRIA,  VA.  22314 

1 

L.I.  Chasen 

P.  O,  Box  8555 

Philadelphia,  Pa.,  19101 

1 

A.  A.  Aibanese 

CSP  4-57,  Syracuse,  NY  13221 

1 

R.J.  Anderson 

CSP  3-35,  Syracuse,  NY  13221 

1 

C.  Arabadjis 

CSP  3-16,  Syracuse,  NY  13221 

1 

R.  L.  Benfey 

CSP  5-C4,  Syracuse,  NY  13221 

1 

C.E.  Blom 

CSP  5-M8,  Syracuse,  NY  13221 

*1 

W.  C.  Bookheimer 

FRP  1-6D,  Syracuse,  NY  13221 

1 

R.  W.  Bush 

CSP  3-11,  Syracuse,  NY  13221 

1 

M.  M,  Clark 

CSP  5-J2,  Syracuse,  NY  13221 

1 

K.  B.  Crocs 

CSP  3-1.1,  Syracuse,  NY  13221 

2 

B.  A.  Deresh 

CSP  5-C4  Syracuse,  NY  13221 

1 

R.Jr  Drexler 

CSP  5-2J,  Syracuse,  NY  13221 

1 

W.  I,  Fersler 

CSP  5-2J,  Syracuse,  NY  13221 

1 

M.  M.  Fitelson 

FTP  1-10B,  Syracuse,  NY  13221 

1 

M.I.  Fox 

FRP  1-ID,  Syracuse,  NY  13221 

1 

E.J.  Gersten 

CSP  5-K4,  Sy  acuse,  NY  13221 

« 

i 

E.  H.  Gibbons 

CSP  1-8,  Syracuse,  NY  13221 

1 

J.  J.  Gostin 

CSP  3-50,  Syracuse,  NY  13221 

1 

W.  D.  Haynes 

CSP  5-2J,  Syracuse,  NY  13221 

1 

J.  F.  Jaeger 

FRP  1-6C,  Syracuse,  NY  13221 

1 

J.K.  Jamison 

CSP  4-57,  Syracuse,  NY  13221 

1 

H.  L.  Johndrow 

CSP  5-G7,  Syracuse,  NY  13221 

1 

J.  F.  Jones 

CSP  3-11,  Syracuse,  NY  13221 

MESO  4034  REV.  e/so 


JiV  V  V.  V.CT  %’  V."  V  V 


1  •.* 


Ccaies  Name  Location 


1 

B.A.  Loomis 

CSP  5-J2,  Syracuse,  NY  13221 

1 

J.  S.  Mac  Blane 

CSP  5-J2,  Syracuse,  NY  13221 

i 

T.A.  Mataumoto 

CSP  1-20,  Syracuse,  NY  13221 

i 

C.A.  Miglierlna 

CSP  3-35,  Syracuse,  NY  13221 

i 

G.H.  Millman 

CSP  5-4B,  Syracuse,  NY  13221 

i 

A.  E.  Morris 

CSP  5-2J,  Syracuse,  NY  13221 

l 

D.J.  Morrow 

CSP  5-K7,  Syracuse,  NY  13221 

l 

K.  A.  Olsen 

CSP  4-41,  Syracuse,  NY  13221 

l 

J.  I..  Perry 

CSP  5-K7,  Syracuse,  NY  13221 

i 

J.E.  Phillips 

CSP  5-W7,  Syracuse,  NY  13221 

l 

P.  E.  Posteli 

CSP  5-K7,  Syracuse,  NY  13221 

l 

E.  L.  Post 

CSP  5-7G,  Syracuse,  NY  13221 

l 

W.D.  Putman 

CSP  5-5P,  Syracuse,  NY  13221 

i 

D.  T.  Rakoske 

CSP  5-2J,  Syracuse,  NY  13221 

l 

J.G.  Reddeck 

CSP  4-5,  Syracuse,  NY  13221 

l 

G.V.  Richards 

CSP  5-K7,  Syracuse,  NY  13221 

l 

E.B.  Rockwood 

CSP  5-2J,  Syracuse,  NY  13221 

l 

R.  A.  Scfcirmer 

CSP  3-11,  Syracuse,  NY  13221 

l 

F.  Schlect 

CSP  5-G7,  Syracuse,  NY  13221 

2 

E.J.  Schroeder 

CSP  5-K7,  Syracuse,  NY  13221 

1 

T.  B.  Shields 

CSP  5-K7,  Syracuse,  NY  13221 

1 

F.  V.  Telllon 

FRP  2-14R,  Syracuse,  NY  13221 

1 

R.  K.  Urquhart 

CSP  3-35,  Syracuse,  NY  13221 

1 

R.  Wasiewicz 

CSP  4-5,  Syracuse,  NY  13221 

1 

R.  E.  Wengert 

CSP  1-20,  Syracuse,  NY  13221 

1 

R.L.  Winje 

FRP  1-6D,  Syracuse,  NY  13221 

1 

A.  E.  Zebrcwski 

CSP  5-K7,  Syracuse,  NY  13221 

> 

±1 


u 

I 


*.T.lTv 


r,r. r. r.  r.  r.  i-.r.^^l,  r.  J-.  w.  .-  V.V.'WA.V.^V.V'.1. 


TABLE  OF  CONTENTS 


Section 


Title 


Page 


INTRODUCTION 

1. 1  Summary 

UNBIASED  OPTIMAL  ESTIMATION  OF  VARIANCE 

2. 1  Deriving  the  Unbiased  Form  for  Known  Polynominal  Order  K 
SUBOPTIMA L  ESTIMATION  OF  VARIANCE 

3. 1  Urderfitting  with  Order  M<K 

3. 2  Overfitting  with  Order  M>  K 

ADJACENT  PO!NT  AUTOCORRELATION  AS  AN  INDICATOR 
OF  OVERFITTING 


4. 1  Concept 

4. 2  Analysis 

REDUCING  ESTIMATION  ERROR 


5. 1  Analysis 
MISSING  DATA  POINTS 

6.1  Orthogonal  Polynomials  for  Missing  Data  Points 

6.2  Computational  Equations 

STATISTICS  OF  THE  VARIANCE  ESTIMATE 


7. 1  Polynomial  Assumption 

7. 2  Distribution  and  Moments  of  the  Estimate 

7. 3  Overfitting  the  Trend 

7.4  Insufficient  Sample  Size 

7. 5  Uncertainty  in  the  Estimate  of  Standard  Deviation 
STATISTICAL  RISK  AND  CONFIDENCE  BOUNDS 

8. 1  Distribution  of  the  Estimate  Accession  For _ 

8. 2  Producer's  Risk  ""ntis  GRAAI 

8.3  Buyer's  Risk  dtic  tab  i 

8.4  Confidence  Bounds  v- -announced 


1  Justification - 


REFERENCES 


j  Distribution/ 

!  Availability  Coco: 
|Aveil  and/or 
Ui st  .  |  Special 

\,U\\  I 


*. '  V  *  '  +  '  >  r- r-  r-  V’-T- 


TABLE  OF  CONTENTS  (CONT) 


Section 

Title 

Page 

APPENDIX  A 

DERIVATION  OF  THE  TRACE 

A-l 

APPEND  DC  B 

DISTRIBUTION  FOR  THE  PARTITIONED  DATA  SET 

B-l 

APPENDIX  C 

MONTE-CARLO  SIMULATIONS  TO  VALIDATE  THE  TECHNIQUE 

C-l 

APPENDIX  D 

SAMPLE  RESULTS  FOR  A  REAL  FLIGHT  TEST 

D-l 

APPENDIX  E 

STATISTICAL  COMPARISON  OF  RESULTS  FROM  A  REAL-FLIGHT 
TEST  WITH  A  CO-LOCATED  PRECISION  REFERENCE  STANDARD 

E-l 

APPENDIX  F 

AN  EXAMPLE  OF  THE  INVERSE  CHARACTERISTIC  FUNCTION 

BV  DISCRETE  FOURIER  TRANSFORMS 

F-l 

LIST  OF  ILLUSTRATIONS 


Figure 

1-1 

1-2 

1-3 

1-4 

3-1 

3-2 

3-3 


Title 

A-tual  PPI  Tracks  of  Lear  Jet  Legs 

PPI  For  Fighter  at  38  kft  Nominal 

Rms  Azimuth  Accuracy  for  Fighter  at  38  kft 

Rma  Range  Accuracy  for  Fighter  at  17  kft 

Mersrred  Data  (Gaussian  Noise  Plus  Zeroth  Order  Trend) 

Residuals  for  M  =  N-2 

Numerical  Example  of  Variance  Estimates  with  Overfitting 
(N  =  10,  K  =  0) 

Measured  Data  and  Residuals  with  an  Underfit  Trend 

Measured  Data  and  Residuals  with  an  Overfit  Trend 

Ratio  of  Normalization  Factors  for  Variance 

Range  History  for  Cartesian  Linear  Flight  Path  with  10-nmi 
North  Offset  and  Asymptotic  Azimuth  of  30° 

Azimuth  History  for  Cartesian  Linear  Flight  Path  with  10-nmi 
North  Offset  and  Asymptotic  Azimuth  of  30° 


Page_ 

1-3 

1-3 

1-4 

1-5 

3-5 

3-5 


il 


n"\r*  .*\1  v 


7," -t  i’.T- 


:  *■  r:  r- 1  ■, 


LIST  OF  ILLUSTRATIONS  (CONT) 


* 

V 

i 

y 


Figure 

Title 

Page 

8-1 

Producer's  Risk  of  Falsely  Falling  an  Acceptance  Test 

8-5 

8-2 

Number  of  Degrees  of  Freedom  for  a  8th  Order  Polynomial  Trend 

8-7 

8-3 

Buyer's  Risk  of  Falsely  Passing  an  Acceptance  Test 

8-8 

8-4 

Distribution  of  the  Variance  Estimate  Showing  the  Confidence  Bounds 
of  Interest 

8-9 

8-5 

Producer's  Risk  per  Test  vs  Total  Number  of  Tests  with  Overall 

Risk  of  10%  and  the  Number  of  Permitted  Failures  as  a  Parameter 

8-12 

8-6 

Buyer's  Risk  per  Test  vs  Total  No.  of  Tests  with  Overall  Risk  of  10% 
and  the  No.  of  Permitted  Failures  as  a  Parameter 

8-12 

iii/iv 


V 

< 


a 

a 

i 

1 

d 

«N 

S 


£4 


1 


‘•'H 


•I 


-H 


'■I 


:4 


>_v.  >-V 


SECTION  I 


INTRODUCTION 

Estimation  of  the  root-mean-square  (rms)  target  position  accuracy  of  a  radar  system 
is  of  great  importance  to  both  radar  manufacturers  and  customers.  To  determine  whether 
a  system  meets  specifications,  tests  are  performed  on  real  targets  and  position  error  is 
estimated  from  data  which  is  stochastic  in  nature,  and  reflects  aircraft  deviations  from  the 
expected  flight  path  as  well  as  from  sources  of  error  within  the  radar  itself. 

For  a  3-D  radar,  ooe  would  like  to  estimate  the  rms  error  in  each  of  three  coordin¬ 
ates:  range,  height,  and  azimuth.  Three  major  components  contribute  to  this  error  in  each 
case.  The  first  of  these  is  bias,  which  is  usually  highly  correlated  from  sample  to  sample, 
but  has  an  error  component  which  is  random  over  the  long  run  in  addition  to  a  fixed  com¬ 
ponent.  This  error  can  only  be  characterized  with  reference  to  some  external  standard. 

The  second  and  third  components  of  error  are  jitter  and  thermal  noise.  These  are  both 
random  errors  which  are  uncorrelaled  from  sample  to  sample.  Their  combined  contribu¬ 
tion  can,  m  principle,  be  characterized  without  need  for  an  external  standard.  The  jitter 
component  is  due  to  a  variety  of  independent  random  sources  and  can  usually  be  treated 
accurately  as  Gaussian  and  Independent  of  target  range.  The  contribution  of  range-dependent 
thermal  noise  is  also  Gaussian,  and  because  the  coordinate  estimation  processes  are  only- 
weakly  nonlinear,  the  sum  of  these  effects  is  still  effectively  Gaussian.  It  is  the  variance 
of  these  random  components  that  we  wish  to  estimate  as  zero  mean  processes,  thus  requiring 
a  separation  of  the  mean  flight  path  from  the  measurement  errors. 

The  traditional  procedures  for  determining  positional  accuracy  consist  of  a  series  of 
controlled  flights.  These  generally  involve  an  aircraft  flying  an  approximately  radial  flight 
path  (with  respect  to  the  radar)  at  a  constant  altitude.  The  intent  here  is  to  control  the  target 
radar  cross  sec  ion  (RCS)  and  to  permit  simplified  flight  path  models.  Usually  two  or  more 
legs  are  run  (e.  g.  inbound  and  outbound)  at  each  of  several  altitudes.  Quite  often  little  or 
no  thought  is  given  to  the  quality  of  the  measured  and  estimated  quantities  in  terms  of  the 
effect  of  limited  sample  sizes  on  estimation  error.  Both  the  producer  and  the  buyer  thus 
suffer  unknown  risks  of  the  radar  falsely  failing  the  test  or  falsely  passing,  respectively. 
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The  total  rms  error  (bias  and  random)  of  the  radar  under  test  can  be  estimated  by 
reference  to  the  simultaneous  track  of  a  precision  tracking  radar  with  known  errors  which 
are  significantly  less  than  the  radar  under  test.  Alternatively  the  error  can  be  separated 
into  two  parts,  with  bias  error  determined  through  static  tests  or  other  means  and  the 
random  error  estimated  from  least  squares  fits  of  the  tracking  data  to  some  simple  trend 
model  (e.g. ,  zeroth-order  in  height  and  azimuth  and  linear  in  range  for  a  radial  flight 
path). 


Several  practical  problems  are  immediately  evident  when  analyzing  data  from  flight 
teats.  First,  if  the  tests  are  manually  conducted  using  an  operator-controlled  track  ball/ 
target  window  on  the  plan  position  indicator  (PPI)  for  instance,  operator  error  can  be  a 
significant  contribution  to  the  total  error.  This  is  especially  true  near  Jie  limits  of  cover¬ 
age  and  detection,  in  the  vicinity  of  crossing  aircraft,  and  in  regions  of  clutter  lea*  through. 
Second,  deviations  from  a  true  radial  flight  path,  including  offsets  and  perturbations  as  from 
gusts,  can  cause  severe  nonlinearities  with  equivalent  polynomial  orders  of  fit  as  high  as  ten 
or  more,  particularly  when  the  target  is  near  the  radar.  Because  of  these  reasons,  simple 
a  priori  models  can  lead  to  position  errors  many  times  larger  than  the  contribution  from  the 
radar  system  alone. 

Often  the  radar  will  be  tested  in  a  location  which  precludes  the  availability  of  a  separ¬ 
ate  reference  radar  and  the  second  approach  is  the  only  practical  alternative.  Accordingly, 
this  paper  addresses  the  problem  of  estimating  the  variance  of  the  random  error  component 
without  a  precision  reference  standard,  providing  a  systematic,  rigorous  approach  which 
is  unbiased  in  the  mean  and  independent  of  the  true  aircraft  flight  path. 

The  method  is  quite  general  to  sensors  and  targets  of  all  types  even  though  the  specif’ 
application  it  was  developed  for  was  a  radar  under  test  with  an  aircraft  on  a  radial  path. 

The  statistics  of  the  estimate  are  known  and  the  concept  of  risk  can  be  used  to  ef¬ 
fectively  design  the  test  (e.g. ,  number  of  legs)  ir.  the  first  place,  while  the  concept  of  con¬ 
fidence  bounds  can  be  used  to  effectively  evaluate  the  test  results  afterwards. 


Figures  1-1  and  1-2  are  PPI  plots  of  actual  radar  data  taken  from  a  modern  tactical 
radar.  They  illustrate  some  of  the  difficulties  involved  in  flying  easily  modelled  paths. 
The  target  in  Figure  1t-1  is  a  Lear  jet  attempting  to  fly  radial  flight  paths.  The  actual 
paths  being  flown  are  not  even  linear  in  cartesian  space,  much  less  in  the  space  defined 
by  the  coordinates  of  range  (R),  height  (H),  and  azimuth  (A).  Because  of  this,  a  linear 


model  Is  severely  inadequate  as  a  trend  mode.  The  target  in  Figure  1-2  is  a  military 
tighter  plane,  also  attempting  to  fly  radial  flight  paths.  Here,  the  path  i&  very  close  to 
being  ltnoar  In  cartesian  space,  but  is  still  very  nonlinear  in  radar  space  particularly  at 
dose  range.  Again,  if  simple  flight  path  models  are  used,  they  will  give  rise  to  severe 
estimation  errors. 

As  evidence  of  this  thesis,  Figure  1-3  shows  the  estimated  error  in  azimuth  as  a 
function  of  range  for  one  of  the  flights  in  Figure  1-2.  The  dotted  line  in  this  figure  repre¬ 
sents  the  theoretical  standard  deviation  of  the  random  error  as  predicted  by  theoretical 
models  of  the  radar.  These  models  include  die  effects  of  stepped  (clutter-rejection)  atten¬ 
uators  which  cause  the  discontinuties  in  the  curve.  The  dashed  line  is  the  one-sigma  random 
error  calculated  from  the  flight  data  after  removal  of  the  mean.  This  corresponds  to  an 
assumed  rsdisl  flight  path.  The  solid  line  shows  the  one-sigms  random  error  calculated 
from  the  flight  data  after  the  removal  of  an  assumed  linear  (in  azimuth)  trend.  The  broken 


RANGE  (nmi) 

Figure  1-3.  Rms  Azimuth  Accuracy  for  Fighter  at  38  kft 

line  shows  the  one-sigma  random  error  calculated  by  the  method  proposed  in  this  paper, 
where  the  order  of  the  assumed  trend  is  determined  from  the  data  itself.  Notice  that 
the  constant  azimuth  assumption  leads  to  severe  overestimation  and  even  the  assumption 


1-4 


of  a  linear  trend  produces  relatively  large  estimation  errors.  The  error  estimate  calcu¬ 
lated  by  the  proposed  method,  however,  shows  good  correlation  with  the  predicted  value. 

Figure  1-4  shows  a  plot  of  the  one-sigma  random  error  in  range,  as  a  function  of 
range,  for  a  fighter  at  17,000  ft.  As  before,  error  ca’culations  based  on  a  linear  trend 
appear  as  a  solid  line,  calculations  based  on  the  proposed  method  as  a  broken  line,  and 
predicted  theoretical  error  as  a  dotted  line.  Notice  again  how  poorly  the  linear  trend 
assumption  compares  with  the  proposed  model. 

These  figures  illustrate  the  need  for  a  method  of  variance  estimation  which 
isolates  the  random  errors  in  the  system  from  the  mean  flight  path.  The  method  developed 
here  addresses  this  problem  directly  and,  in  doing  so,  overcomes  many  of  the  pitfalls  of 
the  simpler  methods. 


Figure  1-4.  Rms  Range  Accuracy  for  Fighter  at  17  kft 


1.1  SUMMARY 


Because  the  random  components  of  error  in  all  three  coordinates  (R,  H,  and  A)  are 

closely  represented  by  zero-mean  Gaussian  processes,  it  is  assumed  that  the  analysis 

below  can  be  applied  with  equal  validity  to  any  of  these  parameters.  For  this  reason,  the 

analysis  has  been  carried  out  in  terms  of  the  general  coordinate  Z,  which  can  be  either  R, 

o 

H,  or  A.  It  will  be  shown  that  the  variance  of  the  random  components  of  error,  a  ,  can  be 
estimated  from  incomplete  data  (data  points  missing),  using  a  polynominal  trend  model. 

This  estimate  is  unbiased  and  independent  of  the  trend  removal  process  in  the  mean.  The 
trend  removal  process  is  adaptive,  statistically  sound,  rigorous,  and  independent  of  a 
priori  or  external  data.  It  will  be  shown  that  overfitting  the  trend  gives  the  same  estimate 
of  the  variance  in  the  mean  as  optimal  fitting.  It  will  also  be  shown  that  the  adjacent  point 
correlation  coefficient  can  be  used  as  an  indication  of  overfitting,  thereby  insuring  proper  fit. 

After  developing  the  variance  estimation  procedure,  the  statistics  of  the  estimation 
error  will  be  derived  and  shown  to  be  directly  relatable  to  the  way  in  which  the  test  is 
carried  out.  Then  the  producer's  risk  of  the  radar  falsely  failing  the  test  and  the  buyer's 
risk  of  the  radar  falsely  passing  the  test  will  be  derived  as  a  function  of  the  statistical 
uncertainty  created  by  limited  sample  sizes.  These  are  important  features  of  a  well 
designed  acceptance  test  in  which  both  types  of  statistical  risks  are  quantified  and  con¬ 
strained  to  mutually  agreed  levels.  Additionally  expressions  are  developed  and  algorithms 
presented  for  estimating  confidence  bounds  on  the  true  variance  given  the  actual  estimate. 

In  all  cases,  simulation  has  been  used  to  confirm  theoretical  techniques  in  the  face  of 
real-world  applications.  Also  shown  are  the  results  of  a  comparison  of  this  technique  with 
the  use  of  an  external  standard  of  reference  in  conjunction  with  data  collected  during  actual 
flight  tests. 
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SECTION  II 


UNBIASED  OPTIMAL  ESTIMATION  OF  VARIANCE 
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a.  1  DERIVING  THE  UNBIASED  FORM  FOR  KNOWN  POLYNOMIAL  ORDER  K 


As  a  starting  point,  assume  that  the  actual  polynomial  trend  of  the  mean  target  flight 
path  is  of  known  order,  K,  with  additive  independent  samples  of  Gaussian  noise.  Taking  N 
measured  data  points,  denote 


Z=[ZlZ2....ZN]‘-Pb  +  v 


(2-1) 


as  the  set  of  measured  positions,  e.  g. ,  range,  azimuth,  or  height,  in  a  specified  time  or 
range  interval,  where 

T 


v=[v1v2....vn] 


is  the  set  of  independent  noise  samples  with 


(2-2) 


E(vJ=  0  and  E^v^j  =  a2  6^  . 


(2-3) 


,th 


The  j  point  of  the  true  mean  flight  path  is 

K  +  l 


v£bip 


11 


(2-4) 


i  =  l 


where  the  are  polynomials  with  j  c  N  and  icK  + 1.  The  vector  of  true  coefficients  is 


*-[ 


~  r bn,  * « *  b, 


'1  ~2 - JK  +  l] 


and  the  polynomial  matrix  is 


P  = 

(NxK+1) 


P11  P12’ 


N1 


l.K  +  l 


N,  K  +  1 J 


(2-6) 
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If  a  polynominal  of  order  K  is  assumed  (it  doesn't  matter  whether  it  is  the  same  polynomial 
ao  long  as  it  is  complete  over  the  same  space),  then  the  assumed  trend  is  (2-5) 


|=pfi 

where  by  the  method  of  least  squares, 


(2-7) 


ft  =|\  V  •  •  \+i]  =  (£Tp)_1pTz  • 


Defining  the  set  of  residuals 

z  4  z  A 

and  ignoring  a  weighting  matrix  for  simplicity,  the  maximum  likelihood  estimate 
of  the  variance  tr2  is 

2  1 

a  ML  N  —  —  * 


(2-8) 


(2-9) 


(2-10) 


Inorder  to  examine  the  structure  of  the  estimate,  first  introduce  Equations  (2-1),  (2-7), 
and  (2-8)  into  the  expression  for  the  residual,  i.e. ,  Equation  (2-9).  Thus 


Z  =  d-M)  Z. 

where  _!  is  the  identity  matrix  and 

A  t  -1  T 
M  *P(P  P)  P  . 


(2-11) 


(2-12) 


Note  that  the  expected  value  of  Z  is  zero  so  that  the  residuals  themselves  are  unbiased. 

Introducing  equations  (2-11)  and  (2-12)  into  (2-10),  recognizing  that  M  is  idempotent 

2  T 

(i.a. ,  M  =M)  and  symmetric  (i.e. ,  M  =  M)  then 

A 

o2  =  Tf  vT(I-M)v  .  (2-13) 

ML  w 


Because  this  is  In  quadratic  form  and  the  noise  samples  are  uncorrelated,  the 
expectation  value  Is 

A  2 

E  cr 1  =  §  Trace  (L-M)  . 

ML  w 


(2-14) 


2-2 


<■. 


Note  that  the  dimensions  of  I_  are  NxN  and  that 

Trace  (L-M)  -  N- Trace  P  (P^P)  ^P*^ 
=  N- Trace  <PTP)_1PTP  =  N  "  (K+l) 


because  the  dimensions  of  Pare  NxK+1.  Thus 


„  _  2,N-K-1  . 

E(tr  )  =<r  < — ^— ) 

ML  W 


<2-15) 


which  is  biased  low  for  all  K  >  0.  An  unbiased  estimate  of  variance  can  be  constructed 
however  from 


A  ~  t-\ 

2  Z  Z 

<7  A/  —  - 


where 


N-(K  +  1) 


(2-16) 


E  (a  2)  =  o  2 


(2-17) 


for  all  N>  K  >1. 


This  formulation  reflects  the  equivalent  loss  of  data  points  in  the  estimation  of  the 
K+l  polynominal  coefficients  from  the  same  data  for  which  the  variance  is  estimated. 
The  reader  will  note  the  familiar  form  this  reduces  co  when  K  =  0,  i.  e. , 


<vV2 

_____ 


for  which  the  mean  trend  is  just  the  average  value  of  the  data 


(2-18) 


N 


Z  = 


(2-19) 
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SECTION  in 


SUB  OPTIMAL  ESTIMATION  OF  VARIANCE 

In  general,  the  true  order  of  the  mean  flight  path,  K,  is  unknown.  Computationally, 
an  order,  M,  la  assumed  which  may  oe  less  than  (underfitting)  or  greater  than  (overfitting) 
the  true  order  K. 

3.1  UNDERFITTING  WITH  ORDER  M<K 


Intuitively,  it  is  clear  that  underfitting  is  dangerous  because  actual  deviations  of  the  true 
flight  path  from  the  assumed  flight  path  will  show  up  as  a  bias  in  the  variance  estimate. 

To  tee  this  analytically,  consider  the  estimation  of  the  M  + 1  polynominal  coefficients  and 
the  resulting  residuals.  Thus,  we  partition  all  matrices  at  the  boundary  of  estimatable 
quantities  such  that 


and 


where 


ub  i i  ]- 

(NxM+1)  (NxK-M) 


P 

(NxM+1) 


11 


1,  M+l 


*1,  Mf2*  *  *  *  Pl,  K+l 


N1‘ 


•PN,M+i;  PN,  M+2. . . .  PN,  K+l 


(3-1) 
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K+l 


K • <piT  pi>'1  piT — 


(3-2) 


(3-3) 


and 


(3-4) 


Defining  a  new  matrix 

=  Pi  (P^Pl)  lpiT 


(3-5) 


3-1 
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Z  =  P  b  +  v  -  Pj  =  (I-  Mx)  (P2  b?  +  v) 


(3-6) 


E(Z)  =(I“M1)  P2b2  ft  o. 


(3-7) 


Thus  the  residuals  have  an  incorrec table  bias  error  because  of  the  unknown  true 
coefficients  b2  /  0.  Moreover,  after  substituting  into  Equation  (2-10)  for  the  traditional 
variance  estimate 


A  2  2/(N-M  +  l)\ 

aML  =<r  - ) 

N  i  .  K+i  v  9 

•si  te 


j-1  \j=M+2 


(3-8) 


E  %E  b*p«E  b*p“ 


i  =j+i 


1  =  M+2  i  =  M+2 


it  is  clear  that  while  the  first  term  is  of  the  form  which  has  a  correctable  bias,  the 
remainder  is  hopelessly  structured  with  the  unknown  and  arbitrary  true  polynomial 
coefficients  bfl  ,  Thus,  underfitting  causes  uncontrollable  and  arbitrarily  large  bias 
errors  whichexplain  the  large  errors  seen  in  Figures  1-3  and  1-4*.  Clearly  under¬ 
fitting  is  to  be  avoided  at  all  costs. 


♦While  it  has  not  yet  been  proved  in  general,  simulation  supports  the  notion  that  underfitting 
leads  invariably  to  overestimation.  As  an  example,  consider  the  simplest  case  where  K=1 
and  M=0.  Using  the  unbiased  expression  for  variance,  i.e. ,  Equation  (2-16),  replacing  K 
by  M  and  equally  spaced  data  points  A  t  seconds  apart  then 


such  that 


A  2 

E  (a  )  >  a  c  for  all  N  >  1. 


,2  *(2 
b2  At 
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3.2  OVERFITTING  WITH  ORDER  M  >  K 

Intuitively,  overfitting  appears  dangerous  because  the  polyncminal  fit  to  the  trend  may 
fit  the  noise  we  wish  to  characterize.  Surprisingly,  this  fear  is  not  justified.  Consider  a 
new  partitioning  at  the  boundary  of  true  quantities  such  that 


P 

(NxM+1) 


-[h.  i  b.]  ‘  '■ 

(NxK+l)  (NxM-K) 


1,  K+l 


N,  K+l 


A  A 
A  A 

blb2 


where 


b  =  (pTp)  1  PTZ 


Pl,  K+2  •••*  Pl,  M+l 


PN,K+2  ••••  PN,  M+l 


(3-9) 


(3-10) 


(3-11) 


+  v  -  P  b 


(3-12) 


Substituting  Eauation  (3-11)  into  (3-12)  and  simplifying 


Z  =  (KM)  Z  =  (I_-M)  Pj  bx  +  (I-M)  v, 


(3-13) 


which  is  the  eame  as  Equation  (2-11)  for  the  case  of  M -K  except  for  the  first  term. 
But  since 


(I-M)  P  =  0 


(3-14) 


all  partitions  of  (I_-M)  P  must  be  zero  as  well.  Thus 

(i-M)  Px  b  =  0  (3-15) 

and 

ft 

Z  =  (I_-M)  v,  (3-16) 

which  is  exactly  the  same  result  achieved  for  the  optimal  case  where  the  polynomial  order  is 
known  beforehand.  Thus  overfitting  leads  to  unbiased  residuals  and  an  unbiased  estimate  of 
variance  can  be  constructed  from 


(3-17) 


ft 

Z  =  (£-M)  Z  (3-18) 

where  the  hat  implies  suboptimization  in  the  sense  that  K  is  not  known  but  M  >  K  is  somehow 
guaranteed  (i.  e. ,  overfitting). 


The  surprising  result  of  this  analysis  is  that  an  unbiased  optimal  e*  Mmate  of  variance 
can  be  achieved,  i.  e. , 


for  all 

K  <  M  <  N-l.  (3-20) 


Intuitively,  it  seems  as  though  overfitting  merely  fits  the  noise.  If  M  =  N-l  then  this 
notion  is  correct.  However,  go  long  as  M  <  N-2  the  trend  does  not  quite  fit  the  noise 
(although  it  is  influenced  by  it),  and  the  variance  estimation  formula,  i.e.  Equation  (3-17) 
exactly  scales  the  residuals  in  the  mean  to  their  correct  value  just  as  if  M  =  h. 


Figures  3-1  and  3-2  illustrate  this.  The  dashed  curve  in  Figure  3-1  shows  simulated 
noise  as  fit  by  the  curve  when  M  =  N-l  and  K  =  0.  The  solid  curve  represents  an  overfit 
where  M  =  N-2.  Note  that  while  influenced  by  the  noise,  there  are  still  finite  residuals  as 
shown  in  Figure  3-2.  These  residuals  are  exactly  scaled  in  the  mean  to  their  optimal  value, 
i. e.  M  =  K,  by  the  factor  ^N/(to-M-I). 
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Figure  3-1.  Measured  Data  (Gaussian  Noise  Plus  Zeroth 
Order  Trend) 
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Figure  3-2.  Residuals  for  M  =  N-2 
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To  fix  ideas  about  the  variance  estimate,  consider  Figure  3-3  which  shows  the  effect 
of  residual  scaling  for  N  =  10,  K  *  0,  a  -  1  as  a  function  of  M,  the  assumed  order.  The  Lower 
curve  on  the  left  shows  the  standard  deviation  (root  variance  estimate)  without  scaling.  Here 
is  the  support  for  our  intuition  of  decreasing  variance  until  M  =  N-l  where  the  noise  in 
exactly  fit.  The  upper  curve  on  the  left  shows  that  the  bias  correction  factor  ^rJ/(N-M-l) 
exactly  scales  the  residuals  for  all  id  <  N-l. 


Figure  3-3.  Numerical  Example  of  Variance  Estimates 
with  Overfitting  (N  =  10,  K  =  0) 

The  plot  on  the  right  shows  the  variation  of  the  bias  correction  factor  as  a  function 
of  M  (for  K=0).  Here  it  cun  oe  seen  that  this  correction  factor  which  scales  the  residuals 
must  increase  as  M  is  increased  over  K  until  at  M=N-1  this  factor  must  be  unrealizably 
infinite  for  the  corresponding!}  zero  residuals.  The  message  here  is  that  so  long  as 
K<M<N-2  the  correction  factor  works  and  unbiased  estimates  of  variance  will  result. 
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SECTION  IV 

ADJACENT  POINT  AUTOCORRELATION  AS  AN  INDICATOR  OF  OVERFITTING 

It  was  shown,  on  the  one  hand,  that  underfltting  leads  to  uncorrectable  bias  errors 
while  overfitting,  on  the  other  hand,  will  always  yield  an  unbiased  estimate  so  long  as 

N-l  >  M  >  K  .  (4-1) 

This  result  is  only  useful  when  accompanied  by  a  reliable  method  for  ensuring  that  M  >  K. 

As  it  happens,  a  sensitive  indicator  of  overfitting  is  the  adjacent  point  correlation  coefficient 
defined  by 
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(4-2) 


where  all  definitions  are  as  before  except  for  the  lag  matrix  defined  by 
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(4-3) 


4.1  CONCEPT 

Underfitting  results  in  residuals  which  are  positively  correlated  because  adjacent  pairs 
are  mostly  of  the  same  sign.  This  effect  is  illustrated  in  Figure  4-1.  Moreover,  the  greater 
the  underfit,  the  more  positive  the  correlation.  Overfitting,  on  the  other  hand,  results  in 
n  siduals  which  are  negatively  correlated  because  adjacent  pairs  are  mostly  of  opposite  sign. 
This  effect  is  illustrated  in  Figure  4-2.  The  greater  the  overfit,  the  more  negative  the  cor¬ 
relation.  Only  In  the  limit  of  M  =  N-l  does  the  correlation  coefficient  reach  -1. 

A 

Thus,  one  can  expect  that  the  mean  value  of  p  will  become  negative  when  M  just  exceeds 
K.  Of  course,  it  isn’t  really  necessary  to  determine  this  point  precisely  because  the  variance 
estimate  will  still  be  unbiased  no  matter  how  much  M  exceeds  K  (except  for  M  =  N-l).  This 
provides  a  cushion  to  protect  against  the  fluctuations  in  the  estimate  of  ^  . 
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TREND 


Figure  4- 1.  Measured  Data  and  Residuals  with  an  Underfit  Trend 


Figure  4-2.  Measured  Data  and  Residuals  with  an  Overfit  Trend 


4.2  ANALYSIS 


The  expectation  value  of  p  from  Equations  (3-18)  and  (4-2)  when  M  >  K  is, 

Trace  (U  M) 


A  1 
E  (p)  =  — 1 


N-(M+1) 


Trace 


N-(M+1) 


(4-4) 


where  we  note  that  Trace  U  =  0  and  (I^M)  is  idempotent.  From  the  definition  of  M , 

Trace (U M)  =  Trace (pTp)  U  P PT  . 

If  P  is  a  complete  set  of  orthogonal  polynomials  over  domain  N  then 
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and  P  P  is  diagonal.  Thus, 
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P  P  + 
*'  2k 


k=l 


and 


N 

E  pj2k 

1=1 


|  P  P 

rN-l,  k  Nk 


A 

E  (P) 


N-l 

K+l  1 

E 

1  , 

.  1=1 

N-M-! 

1+1,  k 


N 

E  *4 

1=1 


(4-5) 


(4-6) 


(4-7) 


(4-8) 


Without  formal  proof,  laborious  analyses  for  M  =  0,  1,  and  2,  and  numerical  simulations 
for  M  as  high  as  20  establish  that 
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E  (p)  = 


M+l 
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N-l  >  M  >  K  . 


(4-9) 
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This  remarkable  result  la  independent  of  the  choice  of  polynomial,  orthogonal  or  otherwise, 
so  long  as  it  is  complete  over  the  domain  of  N. 

Of  oourse  Equation  (4—2)  is  not  a  form  suitable  for  calculation  because  of  the  need  to 
2 

knew  o  ,  the  true  variance,  beforehand.  A  practical  expression  for  this  purpose  which  has 
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SECTION  V 

REDUCING  ESTIMATION  ERROR 


Because  the  rocximum  Likelihood  estimate  Is  the  minimum  variance  estimator  of  variance, 
the  estimation  error  wilt  be  increased  by  the  bxr  j  correction  factor  N/(N-M-1).  A  practical 
way  of  reducing  this  error  is  to  use  a  larger  data  set  for  the  trend  removal  than  for  the 
variance  estimate. 

5.1  ANALYSIS 

Consider  the  partitioning  of  the  N  data  points  for  trend  removal  into  three  parts,  the  cen¬ 
tral  one  being  used  for  the  estimate  of  variance.  Thus, 


T 1  T  *  T 

T 

1  1 

h  \h  '  V 

Z1Z2*  ‘  *  *  Zr  '  Zr+1  •  ‘  *  Zs  1  Zb+1-  ’  *  ZN 

L  •  t  J 

and  Pis  similarly  partitioned  horizontally  as 


P 

(N  x  K+l) 


T 


(r  x  K+l)  (s-r  x  K+l)  (N-s  x  K+l) 


Then 


a  2  1_ 

°ML  =  s-r 
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From  Appendix  A  where  the  polynomials  are  assumed  to  be  orthogonal, 


(5-5) 


(5-6) 


(5-6  A) 


(5-6B) 


(5-6C) 
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so  that  an  unbiased  estimate  of  variance  for 


s-r  <  N 
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a2  A 


~2  —2 


M+l 


8 


-E 


nk 


n=r+l 


(5-7) 


k=l 


N 

s 

n=l 


nk 


which  reduces  to  the  prior  result,  i.  e.  Equation  (3-17)  for  s-r  =  N.  To  see  that  the  un¬ 
certainty  in  the  variance  estimate  has  been  decreased  by  extending  the  domain  of  the  fit  to 

2/_/a  9  \ 

the  trend,  consider  that  the  bias  correction  factor  a  J  is  much  less  for  a  given  number 

of  data  points  s-r.  Thus  if  s-r  data  pointb  are  used  for  the  variance  estimate  in  either  case, 
the  ratio  of  bias  correction  factors  with  N  >  s-r  and  N  =  s-r  is, 


s-r  -  (M+l) 


<  1  . 


M+l 
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This  is  so  because 
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Figure  5-1  is  a  plot  of  this  ratio  for  s-r=10  and  N=30  for  0  ^  M  <  9  with  K-0.  Notice 
that  for  M=9,  the  numerator  goes  to  zero  because  in  the  case  where  L=N=30,  N-l=9  and  all 
points  are  precisely  fit.  As  discussed  in  Section  III,  the  residuals  are  scaled  exactly  to 
truth  in  the  mesn  for  all  cases  where  M  <  N-l.  Thus  we  see  here  another  advantage  to  in¬ 
creasing  N  over  L  in  that  it  permits  higher  order  fits  without  fitting  all  the  points.  This  is 
offset  somewhat  however  in  that  the  larger  N  is  (with  a  given  spacing), the  larger  M  has  to  be 
in  general.  Nevertheless  the  advantage  still  accrues  for  many  cases  of  interest.  Later 
in  par,  7.3,  it  will  be  shown  that  this  ratio  is  precisely  the  ratio  of  the  variances  of 
the  estimates  for  these  two  cases. 
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SECTION  VI 
MISSING  DATA  POINTS 


The  previous  development  implied  evenly  spaced  data  at  every  one  of  the  N  points.  Be¬ 
cause  this  procedure  is  to  be  used  with  real-world  equipment  with  a  probability  of  detec¬ 
tion  less  the  unity,  it  is  clear  that  variance  estimates  will  be  required  in  regions  having 
missing  data  points. 


6.1  ORTHOGONAL  POLYNOMIALS  FOB  MISSING  DATA  POINTS 


Define  a  vector  H  with  dimension  N'  that  consists  of  N  ones  for  N  detections  and  N'  -  N 
zeros  for  missed  detections, 


HAfH^.  .  .  .  Hn,J 


with 


N' 


v  =  ;  r  *  S  hj  ; 

1=1  1=1 


s' 

1=1 


where  the  relationships  between  r  and  r’  and  s  and  s'  are  the  same  for  N  and  N\ 


Moreover,  define  a  quasi-identity  matrix  _I*  with  H  as  its  diagonal,  i.  e. 


I»  A 


H, 


H, 
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Then  the  variance  estimate  becomes, 

rp  ^ 

Z>  A  Z’ 


a2  = 


Trace  ^ 
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where 


Z*  =  r  Z 


(6-5) 


and 
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Pnk  Hn 


(6-6) 


The  polynomial  set  is  orthogonal  in  domain  N*  with  weighting  matrix  H .  The  orthogonality 
relation  is 


n=l 

The  recursion  relation  for  these  polynomials  is 

Vh  -  (J-1-  M  Vrk  W  2^M 

with 
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(6-10) 


6-2 


and 
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N»  !  N* 
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1=1  /  i=i 


PjX  =  1,  for  all  J  . 

Details  of  this  set  of  orthogonal  polynomials  and  its  use  can  be  found  in  Ref.  (1). 


6.2  COMPUTATIONAL  EQUATIONS 

These  matrix  forms  of  the  equations  for  estimating  variance  from  an  incomplete  data 
set,  l.e. ,  which  Include  missing  data  points,  have  the  following  algebraic  equivalents  for 
computational  purposes: 

£  \  2 
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while  Trace  is  given  by  Equation  (6-6)  and 


M+l 


k=l 


(6-14) 


with 


N' 


A 

b. 
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i=i 
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Z4 
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SECTION  vn 


STATISTICS  OF  THE  VARIANCE  ESTIMATE 

It  was  shown  earlier  that  the  variance  estimate  is  unbiased;  that  is  the  expectation  value 
of  the  estimate  is  the  true  variance  of  the  noise.  Nevertheless  the  estimate  itself  is  a  random 
variable  subject  to  uncertainty  which  arises  primarily  from  three  sources: 

1.  The  assumption  of  a  finite  polynomial  trend 

2.  Overfitting  the  trend 

3.  Insufficient  sample  size  * 

This  section  explains  these  sources  of  uncertainty  and  offers  practical  approaches  to  con¬ 
straining  them  which  have  already  been  proven  both  by  simulation  and  application  to  actual 
test  flights. 

7.1  POLYNOMIAL  ASSUMPTION 

Even  a  cartesian- linear  flight  path  which  doesn't  overfly  the  radar  will  be  trigonometric 
in  the  radar's  polar  coordinate  system.  Figures  7-1  and  7-2  show  the  range  and  azimuth 
histories  of  such  a  flight  path  with  north  offset  of  10  nmi  and  asymptotic  azimuth  of  30°.  In 
principle  a  polynomial  fit  to  this  data,  particularly  at  close  range  would  require  an  infinite 
number  of  terms.  In  practice,  equivalent  polynomial  orders  less  than  eight  or  nine  are 
usually  sufficient  to  fit  mean  flight  paths  down  to  a  small  fraction  of  the  radar  noise  so  long  as 
two  conditions  are  satisfied. 

1.  No  range  interval  for  estimating  the  mean  target  path  extends  beyond  the  point 
of  closest  approach. 

2.  These  intervals  must  be  adjusted  downwards  in  length  to  accommodate  the 
increasing  curvature  (in  radar  coordinates)  encountered  near  the  radar. 

Both  conditions  stem  from  the  fact  that  the  equivalent  polynomial  order  increases  rapidly 
in  the  vicinity  of  the  point  of  closest  approach.  An  interval  which  straddles  this  point  will 
require  a  much  higher  order  of  fit  than  one  which  places  it  at  its  boundary;  thus  the  first 
condition.  Nevertheless  the  reader  is  cautioned  that  this  innermost  interval  may  have  to  be 
broken  up  further,  i.  e. ,  reduced  in  range  extent,  if  the  equivalent  polynomial  order  of  fit 
turns  out  to  be  too  high  for  the  computer  being  used.  A  500-knot  target  with  a  radar  having  a 
revisit  time  of  12  seconds  (5  rpm)  will  have  about  6  data  points  in  a  10-nmi  range  interval. 
Experience  shows  that  a  computer  with  32-bit  floating  point  capability  can  easily  handle  the 
10-15**1  order  polynomials  sometimes  required  in  the  innermost  10-nmi  range  interval. 
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The  second  condition  addresses  the  need  to  optimize  the  number  of  data  points  used  in 
the  fit,  N,  and  the  assumed  polynomial  order  of  fit,  M,  for  a  given  set  of  data  used  for  the 
variance  estimate,  L.  The  total  flight  path  is  usually  divided  into  a  number  of  Intervals  con¬ 
taining  6-  to  10-data  points  a  piece,  1.  e.  L,  starting  with  the  point  of  closest  approach  to  the 
radar.  But  there  are  two  competing  sources  of  error  which  require  disparate  optimal 
strategies  for  picking  N  and  M.  On  the  one  hand,  we'd  like  N  as  small  as  possible  near  the 
radar  to  minimize  the  true  order  of  fit  K,  and  thus  avoid  possible  bias  error  from  inadvertant 
underfitting  in  the  region  where  the  finite  polynomial  assumption  begins  to  breakdown.  On 
the  other  hand,  it  will  be  shown  in  par.  7. 3  that  increasing  N  for  a  given  L  and  true  order  K 
will  result  in  a  dramatic  decreaso  in  the  estimation  uncertainty.  After  many  simulations  and 
the  processing  of  much  real  data,  the  following  strategy  was  derived  which  appears  to  provide 
a  robust  balance  between  these  seemingly  disparate  requirements. 

1.  The  order  of  fit  M  is  increased  from  unity  to  the  maximum  permitted  by  the 
computation  system  with  accuracy,  and  all  those  cases  where  the  correlation 
coefficient  is  negative  for  M,  M-l  and  M-2  are  accepted  as  legitimate 
variance  estimates.  The  requirement  that  p  be  negative  for  M,  M-l  and  M-2 
has  been  shown  by  experience  to  effectively  eliminate  the  effects  of  sampling 
fluctuation  on  the  instantaneous  estimate  p  (M;N). 

2.  For  ranges  less  than  4LAR  (where  A  R  is  the  distance  between  data  points) 

N=L  and  we  pick  the  order  of  fit  M  to  be  the  largest  possible  on  the  computer 
in  use  subject  to  the  additional  constraint  on  the  correlation  coefficient  that 

p  be  negative  for  M+l  as  well  to  provide  additional  assurance  that  the 
polynomial  approximation  is  valid. 


3.  For  ranges  larger  than  4LAR,  N  is  systematically  increased  from  L  to  4L 
and  M  increased  from  the  minimum  established  in  step  (1)  to  the  maximum 
permitted  by  the  accuracy  of  the  computer  (e.  g.  10-15).  The  particular 
values  of  N  and  M  used  are  those  that  correspond  to  the  maximum  Trace  ( ^22  ) 

(as  calculated  by  Equation  (6-6)]  from  the  entire  list  of  legitimate  estimates. 


4.  The  optimum  strategy  regarding  the  geometrical  relationship  between  N  and  L 
is  to  use  symmetry  wherever  possible,  i.e.  for  L  to  be  centered  in  N.  This 
stems  from  the  fact  that  while  all  possibilities  will  lead  to  unbiased  (or  minimally 
biased)  estimates,  the  centered  case  results  in  the  minimum  estimation  un¬ 
certainty.  Near  the  ends  of  coverage  or  the  point  of  closest  approach  L  cannot 
be  centered  and  some  increased  uncertainty  must  be  tolerated. 


Examples  of  the  efficacy  of  these  procedures  are  given  in  Appendices  C  and  D. 
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Basically  the  estimate  of  variance,  as  represented  by  Equation  (6-13)  can  be  viewed  as 
a  process  whereby  s-r  Gaussian,  zero-mean  variates,  i.e. ,  Z^  =  -  Z  ,  are  squared  and 

summed,  and  then  normalized  by  Trace  v  • .  Now  it  is  commonly  known  that  a  random 
variable  which  is  the  sum  of  the  squares  oTn  independent  zero-mean  Gaussian  variates  is  chi- 
aquare  distributed  with  v  =N  degrees  of  freedom.  Moreover  if  the  mean  is  estimated  from 
averaging  the  data,  then  one  degree  of  freedom  is  lost  and  v  =  N-l.  It  is  a  straightforward 
proof  to  show  that  when  M+l  coefficients  are  required  for  the  estimation  of  the  mean  then 
M+l  degrees  of  freedom  are  lost  and  v= N-M-l.  For  L=N  then  it  is  clear  that 
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where  <r  is  the  true  variance  and  v  the  number  oi  degrees  of  freedom  of  the  random 
2 

variable  x  v  .  For  L=N 


a2 x  2 

2  v  . 

a  =  N-M-l 
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Since 
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we  see  that 


E('*)  -  5PffTE(*2)  * 


(7-5) 


which  shows  again  that  the  estimate  is  unbiased,  and 


va*(4)  = 


2a 

N-M-l  ’ 


(7-6) 


which  is  a  new  result. 
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What  happens  when  N  >  L  and  v  =  Trace  ?  With  the  proof  in  Appendix  B,  we  state 
here  that  the  normalized  random  variable  Trace"  *22  /  a  2^ia  8tly  chi-square  distributed 

with  Trace  degrees  of  freedom.  Thus, 

2  2 

/N  C  "Trace  ^ 

a  =  - - -  ”  •  (7-7) 

Trace 
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From  Equation  (7-3), 


P). 
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Trace  ^ 
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Because  Trace  >  s-r-M-1,  as  seen  from  Equation  (5-8),  it  is  easy  to  see  that  the  un¬ 
certainty  in  the  estimate  is  reduced  by  the  procedure  of  using  more  points  for  the  trend 
removal  than  for  the  variance  estimate  itself. 


7.3  OVER  FITTING  THE  TREND 

It  was  already  shown  that  overfitting  does  not  introduce  a  bias.  Nevertheless  there  is 
a  penalty  which  appears  as  an  increased  uncertainty  in  the  estimate.  This  may  be  seen 
clearly  from  Equation  (7-6),  which  is  the  form  of  the  variance  of  the  estimate  when  N=L. 
The  standard  deviation  of  the  estimate  then  varies  almost  inversely  as  the  square  root  of 
order  of  fit,  i.  e. , 


For  N=10,  Table  7-1  gives  the  increase  in  the  standard  deviation  for  the  estimate  with 
as  the  parameter. 


TABLE  7-1 


INCREASED  UNCERTAINTY  DUE  TO  OVER  FITTING  WHEN  N=L=10 


Kd 

0 

1 

2 

3 

4 

5  6  7 

1 

2 

1.14 

1.07 

3 

1.22 

1.15 

1.08 

4 

1.34 

1.26 

1.18 

1.10 

5 

1.50 

1.41 

1.32 

1.22 

1.12 

6 

1.73 

1.63 

1.53 

1.41 

1.29 

1.15 

7 

2.12 

2.00 

1.87 

1.73 

1.58 

1.41  1.22 

8 

3.00 

2.83 

2.65 

2.45 

2.24 

2.00  1.73  1.41 

From  this  table,  the  uncertainty  can  almost  triple  when  the  true  order  is  zero  and  increase 
by  a  factor  of  2  when  the  true  order  is  5. 


The  general  form  of  Equation  (7-9),  where  N  >  L,  is 


SIG 
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SIG 


Trace  y22  (N,  L,  Mn) 


Trace  v ^  (N,  L,  M) 


(7-10) 


As  a  means  of  comparison,  consider  two  examples.  In  both  cases,  we  will  keep  L=10, 
analogous  to  the  example  from  which  Table  7-1  was  constructed.  Tables  7-2  and  7-3,  on 
the  other  hand,  present  the  results  for  N=20  and  40,  respectively,  where  the  L  data  points 
used  for  variance  estimation  are  centrally  located  among  the  N  data  points  used  for  trend 
removal. 
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TABLE  7-2 

INCREASED  UNCERTAINTY  DUE  TO  OVERFITTINS  WHERE  L-10  AND  N-20 


0 

1 

2 

3 

4 

5 

1.007 

1.028 

1.021 

1.051 

1.044 

1.022 

1.067 

1  c060 

1.038 

1.016 

1.092 

1.085 

1.062 

1.039 

1.023 

1.120 

1.112 

1.089 

1.066 

1.049 

1.025 

1.143 

1.136 

1.112 

1.088 

1.071 

1.047 

1.175 

1.167 

1.143 

1.118 

1.100 

1.076 

1.021 

1.049  1.027 


TABLE  7-3 

INCREASED  UNCERTAINTY  DUE  TO  OVERFITTING  WHERE  L-10  AND  N-40 


0 

1 

2 

3 

4 

5 

6 

1.000 

1.015 

1.015 

1.019 

1.018 

1.004 

1.031 

1.030 

1.015 

1.012 

1.039 

1.038 

1.023 

1.020 

1.008 

1.048 

1.047 

1.032 

1.028 

1.017 

1.009 

1.060 

1.057 

1.044 

1.040 

1.028 

1,020 

1.011 

1.067 

1.067 

1.051 

1.047 

1.035 

1.027 

1.018 

It  should  be  evident  from  these  tables  that  the  effect  of  increasing  N  for  a  given  L  are  dra¬ 
matic,.  For  instance,  when  the  true  order  is  zero,  the  uncertainty  barely  tcreases  by  18% 
for  N=20  and  7%  for  N=40.  Compare  this  to  the  300%  increase  shown  in  Ts  le  7-1.  The 
length  of  these  tables  are  kept  to  match  Table  7-1  and  not  out  of  any  theoretical  restriction, 
recalling  that 


K  <  M  <  N-2  (7-11) 

is  the  only  restriction  and  we  could  have  increased  M  to  18  for  Table  7-2  and  38  for  Table 
7-4,  if  needed.  The  point  here  is  that  there  is  very  little  penalty  for  reasonable  amounts  of 
overfitting  with  N  >  2L,  leading  to  a  robustness  for  the  technique.  In  practice  we  have  found 
that  increasing  N  beyond  4L  is  unnecessary  from  the  point  of  view  of  constraining  the  un¬ 
certainty  in  the  estimate. 

7.4  INSUFFICIENT  SAMPLE  SIZE 

Because  a  typical  air  defense  radar  may  only  revisit  a  target  once  every  6  to  12  seconds, 
the  total  number  of  samples  available  for  a  variance  estimate  will  be  small  on  a  single  flight 
leg.  As  an  example,  a  500-knot  target  will  yield  only  6  unique  data  points  in  a  10-nmi  range 
interval  for  a  radar  revisit  time  of  12  seconds.  How  big  will  the  uncertainty  in  the  estimate 
be  ?  From  Equation  (7-8)  we  find  that 

®CT2)-  "2  •  <7’12> 

For  L=6,  N=4L=24,  MQ  =  0,  and  M=8  for  example 


Trace  v2„  =  4.54  . 


Thus 


0.664  a2 


(7-13) 


which  means  that  the  uncertainty  in  the  estimate  of  variance  is  larger  than  65%  of  the  true 
variance  itself.  From  estimate-to-estimate  then  a  considerable  fluctuation  will  be  en¬ 
countered,  perhaps  far  more  that  we  would  like. 
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One  way  to  reduce  this  uncertainty  is  to  extend  the  range  interval,  i.  e.  L,  over  which 
the  estimate  is  made.  This  has  several  disadvantages.  First,  a  large  increase  in  L  may 
force  an  averaging  of  true  variance  which  may  vary  significantly  over  the  range  interval 
because  at  long  range 


Secondly,  the  pressure  increases  greatly  for  a  higher  order  of  fit  than  perhaps  can  be 
handled  with  precision  on  a  typical  computer. 


An  alternative  approach  which  has  proved  quite  effective  in  practice  is  to  increase  the 
cumber  of  flight  legs  for  the  same  (or  similar)  conditions.  Thus  for  p  -  flight  legs  where 
we  average  the  variance  estimates  in  like  range  intervals  after  removing  the  trend  on  each 
leg. 


Note  that  the  result  Is  still  unbiased,  i.e. 


(7-14) 
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For  planning  purposes,  consider  all  N,  L,  and  M  identical  from  leg  to  leg  for  a  given  range 
interval. 


Then, 
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Suppose,  for  the  example  just  considered  we  wish  to  reduce  SIG 


(-1) 


to  only  30%  of  a  . 


Then  from  Equations  (7-13)  and  (7-18),  the  number  of  legs  required  would  be  5,  not  unreason- 

,*2\ 

able.  On  the  other  hand,  44  legs  would  be  required  to  reduce  SIG  I  J  to  30%,  perhaps 
too  many  to  justify  economically. 


This  loads  naturally  into  questions  of  test  planning.  What  value  of  the  ratio 


(%)ls 


SIG  l  j  Is  really  appropriate?  This  is  where  the  concept  of  statistical  risk  is  useful;  a 

subject  treated  in  Section  VIII. 

7.5  UNCERTAINTY  IN  THE  ESTIMATE  OF  STANDARD  DEVIATION 

From  Equation  (7-7)  we  see  (hat  the  variance  estimate  is  a  random  variable  related  to 
die  chi-square  variate  with  v  -  Trace  degrees  of  freedom.  Generalizing  for  p  -  flight 
legs  then 
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where 


v  A  Trace  v » 
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Note  that 
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as  expected. 
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SECTION  VIII 


STATISTICAL  RISK  AND  CONFIDENCE  BOUNDS 


While  the  uncertainties  in  the  estimate  (variance  or  standard  deviation)  are  useful  in  a 
comparative  sense,  they  provide  little  direct  information  helpful  in  designing  tests  or  evalu¬ 
ating  test  results.  On  the  other  hand,  the  concept  of  statistical  risk  is  quite  useful  in  design¬ 
ing  tests  to  both  constrain  the  producer's  risk  of  falsely  failing  and  to  illuminate  the  spectrum 
of  buyer's  risks  of  falsely  passing  (as  a  function  of  his  least  acceptable  performance);  each 
deriving  solely  from  insufficient  sample  sizes.  Moreover,  upper  and  lower  confidence  bounds 
can  be  derived  to  infer  limits  to  truth  implied  by  the  test  results.  Analytical  expressions 
for  these  concepts  are  derived  here  and  numerical  examples  presented  to  illustrate  their  use. 


8.1  DISTRIBUTION  OF  THE  ESTIMATE 

In  the  previous  section,  we  stated  that  the  distribution  of  the  normalized  variance  estimate 
for  one  leg  and  range  interval  was  chi-square  with  v  A  Trace  degrees  of  freedom,  i.  e. , 
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An  estimate  averaged  over  p  legs,  on  the  other  hand,  is  not  in  general  related  to  the  chi- 
square  variable  because 
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Only  when  all  v  =  v  is  the  estimate  clearly  still  related  to  the  chi-square  variable,  i.e. , 
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Returning  momentarily  to  the  general  formulation  of  Equation  (8-3),  we  see  that  the  dis¬ 


tribution  of  t  =  x  /  v  is  a  special  case  of  the  gamma  pdf  ,  i.  e. , 
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where 
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which  has  a  characteristic  function. 
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Because  we  are  dealing  with  a  summation  over  p  such  terms  the  characteristic  function  of 
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which  has  no  known  general  inverse  for  arbitrary  |  0^  j  and  p. 
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There  are  two  reasons  for  needing  the  pdf  in  the  general  case, 

1.  To  recompute  pass/fail  thresholds  (par.  8.2)  and  buyer’s  limits  (par.  8.3)  after 
the  test  when  the  actual  1 j  set  will  have  been  (mown  in  each  range  interval. 

2.  To  compute  relevant  confidence  bounds  to  true  performance  after  the  test  as  inferred 
by  achieved  performance  and  the  actual  ^  j  set. 

In  lieu  of  an  analytical  distribution  (general  inverse  to  Equation  (8-10),  a  histogram  can  be 
generated  numerically  after  the  specific  set  is  known  from  a  suitable  chi-square  random 
number  generator  structured  according  to  the  leg-averaged  variance  estimate,  i.  e. , 

Equation  (8-3).  Another  way  is  to  numerically  perform  the  inverse  to  the  general  charac¬ 
teristic  function  with  a  specific  set;  perhaps  using  a  discrete  Fourier  transform,  i.e. , 
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so  that. 
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k  =  0,  1,  2, 


N-l, 


/\ 

2  2 

where  t  is  a  sample  of  the  /a  random  variable  and  we  evenly  sample  all  of  t-space  at 
least  2N  times.  Note  that  when  all  j  ^  j  are  equal  for  each  leg  that 


0  -iri 


a  = 


(8-13) 


so  that  the  distribution  becomes  gamma  with  pv  =  p  Trace  v ^  degrees  of  freedom.  An 
example  of  this  technique  is  shown  in  Appendix  F. 
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8.2  PRODUCER'S  RISK 

The  producer's  risk  la  conceptually  defined  as  the  probability  of  his  product  falling  a 
test  when  In  truth  it  should  have  passed,  1.  e. ,  falsely  falling.  Thus  in  terms  of  parameters 
of  interest  to  us  here, 

KP-pr[<’pS!  >  <•?  ^"T2]  (8‘14) 

where  o  is  the  pass/fail  threshold  such  that  the  producer  passes  the  test  if 

ff  2<a_2.  (8-15) 

p  -  T 

It  is  a  simple  matter  to  express  this  risk  mathematically  as, 
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Figure  8-1  is  a  plot  of  this  risk  where  all  ^  =  v  and  the  total  number  of  degrees  of  freedom 


pv  =  p  Trace  v' 
44 


(8-17) 


Noting  that  the  abscissa  is  o  T/a 


consider  a  producer's  risk  of  10%.  If  the 


number  of  degrees  of  freedom  is  only  one  then  the  pass/fail  threshold  on  /  2*  ,  i.e. ,  o™, 

V  T 

must  be  almost  65%  more  than  the  true  value  a .  This  is  the  tolerance  required  to  fairly 
accommodate  the  sampling  fluctuation  expected  in  the  estimate  while  still  leaving  the  producer 
with  a  10%  chance  of  falsely  failing.  If  the  buyer  thinks  this  tolerance  (or  discrimination  ratio) 
too  large,  he  can  insist  on  a  larger  sample  size.  In  our  case,  this  translates  to  more  flight 
legs,  i,e. ,  increasing  p  v.  Note  that  for  pv  =  50,  the  pass/fail  threshold  on  /  2  only  has 

to  be  a  little  over  10%  greater  than  the  theoretical  value  a .  ^  P 
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Because  the  paas/fail  threshold  is  always  related  to  the  theoretical  value,  it  is  necessary 
to  make  an  estimate  of  o  prior  to  the  test.  A  reasonable  candidate  is  the  expected  perfor¬ 
mance  given  that  the  product  performs  as  promised.  Thus  the  producer  and  the  buyer  must 
agree  to  this  <y  as  part  of  the  test  package  which  includes  the  producer's  risk,  the  number  of 
flight  legs,  and  other  more  mundane  details  of  the  test,  e.g.  altitude,  speed,  range  interval, 
flight  azimuth,  etc.  It  is  also  important  to  relate  these  details  to  expected  values  for 
v  -  Trace  v .  Figure  8-2  is  helpful  in  this  regard  for  it  is  a  plot  of  v  for  an  eighth  order 
polynomial  fit  against  N  as  the  independent  variable  and  L  as  the  parameter.  Note  that  L  is 
uniquely  dependent  on  the  speed  of  the  aircraft  and  the  revisit  time  of  the  radar  under  test  and 
is  assumed  here  to  be  centrally  positioned  with  respect  to  j  N  {  .  The  test  planner  can  pick 
some  representative  value  for  N2  2L  and  use  the  resulting  v  for  test  planning  purposes.  For 
range  intervals  close  to  the  radar,  the  value  of  v  used  should  probably  be  halved  because  of 
the  increased  pressure  for  higher  order  fits  and  the  asymmetry  of  {  L  (  with  respect  to 
IN  \  . 

8.3  BUYER'S  RISK 

The  buyer's  risk  is  conceptually  defined  as  the  probability  of  the  product  passing  the 
test  at  the  pass/fail  threshold  when  in  truth  it  should  have  failed  at  the  limit  of  his  acceptable 
performance,  i.  e. ,  the  buyer's  limit.  In  terms  of  parameters  of  interest  to  us  here, 
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Where  a  Q  is  the  buyer's  limit  which  must  be  greater  than  both  the  pass/fail  threshold  oT 

and  the  theoretical  or  expected  performance  c  for  the  buyer's  risk  to  be  lower  than  about 

50%.  There  is  an  asymmetry  between  the  two  risks  shown  here  because  it  is  the  producer's 

product  under  test  and  the  buyer's  risk  arises  only  because  of  his  perception  that  <x  ><r_  , 

1  p 

which  can  never  be  known  for  sure.  Once  the  parameters  of  the  test  are  picked,  i.  e. ,  R  , 
p,  E  (Trace  v*  ),  and  a  ,  the  pass/fail  threshold  is  uniquely  determined.  But  the  buyer's 
limit  is  a  parameter  which  can  be  traded  off  against  the  buyer's  perceived  risk.  Figure 
8-3  is  helpful  here  in  seeing  this  point.  For  a  ^,/Og  =  0.  5  as  an  example,  and  pv  =  1  the 
buyer's  risk  will  be  as  high  as  38%,  but  his  penalty  is  100%,  i.e.  =  1.  if  p^  is  in¬ 

creased  to  50,  on  the  other  hand,  his  perceived  risk  becomes  vanishingly  small,  even  though 
his  penality  remains  high.  On  the  other  hand,  if  he  wishes  to  stand  a  risk  of  10%  with  pv  =  50 
then  the  buyer  may  only  have  to  accept  a  radar  which  is  about  16%  worse  (i.  e. ,  1/0. 86-1) 
than  the  test  threshold,  assuming  that  it  passes  the  test.  Compare  this  to  the  100%  penalty 
he  must  accept  in  the  prior  example  where  his  risk  is  almost  zero. 
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8.4  CONFIDENCE  BOUNDS 

The  concepts  of  producer’s  and  buyer’s  risks  are  seen  to  be  useful  in  providing 
tradeoffs  to  the  design  of  effective,  statistically  meaningful  tests,  to  the  degree  one  can 
estimate  expected  performance,  i.e. ,  a ,  prior  to  the  test.  Once  the  test  is  performed,  it 
is  natural  to  want  to  make  inferences  about  the  true  performance  implied  by  the  test  results. 


A  standard  way  of  doing  this  is  to  provide  upper  and  lower  bounds  to  true  performance, 
given  the  test  result,  associated  on  a  one-to-one  basis  with  a  level  of  confidence.  Figure  8-4 
illustrates  these  bounds  for  the  distribution  of  interest,  i.  e. ,  Equation  (8-13).  With  C  the 
confidence  level,  a  numerical  solution  is  sought  for  a  ^  and  a  ^  from 
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Figure  8-4.  Distribution  of  the  Variance  Estimate  Showing  the  Confidence  Bounds  of  Interest 
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It  is  interesting  to  note  that  because  the  distribution  is  asymmetric,  the  bounds  will  be 
asymmetric  as  well.  As  an  example,  consider  pi>  =  10  and  =  1  left.  Then  the  square 

vTp 

root  of  the  95%  upper  and  lower  confidence  bounds  for  true  variance  are 


and 


a  =  2. 16  kft 


a  L  =  0.63  kft. 


As  another  example,  consider  a  tenfold  increase  j.n  pv  to  100  with  still  1  kft.  Then 


cr  =1.26  kft, 
u 


and 


ffT  =  0.856  kft. 
Li 


8. 5  ADDITIONAL  CONSIDERATIONS  IN  TEST  DESIGN 


Often  an  acceptance  test  on  a  radar  may  include  five  to  ten  separate  performance  mea¬ 
sures,  e. g.  detection  probability,  height  accuracy,  range  resolution,  etc.,  several  different 
altitudes,  and  20  or  more  separate  range  intervals.  The  total  number  of  independent  tests 
may  run  up  to  the  hundreds.  If  the  producer's  overall  risk  is  to  be  kept  to  10%,  for  example, 
and  the  criterion  for  passing  is  that  he  must  pass  all  of  say  100  tests,  it  should  be  obvious 
that  his  allocated  risk  per  performance  measure,  per  altitude,  per  range  interval  will  be 
very  small,  i.e, 


1/100 
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=  0.105%. 
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The  tolerance,  i.e.  a, p/a,  will  be  much  larger  than  for  the  single  tests  discussed  in  par. 
8. 2.  Similarly,  the  buyer’s  risk  per  elemental  subtest  will  be  much  higher  than  for  the 
single  test  discussed  in  par.  8. 3. ,  i.e., 

1/100 

rB=(RB)  *  97-7*  ’ 


probably  intolerable  to  most  customers  without  considerable  explanation.  The  reason  for 
the  dramatic  telescoping  of  risks  is  the  buyer's  pass/fail  criterion  for  passing  the  test;  that 
is  the  producer  must  pass  each  and  every  elemental  subtest  to  pass  overall.  A  slight  loosen¬ 
ing  of  this  criteria  can  pay  dividends  here.  Suppose,  for  instance,  that  the  producer  can  fail 
one  or  more,  say  m,  subtests  and  still  pass  overall.  Mathematically  this  is  expressed  as 
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where  R  and  R  are  the  overall  test  risks,  r  and  r  are  the  elemental  subtest  risks,  N 

8 

is  the  total  number  of  subtests,  and  m  is  the  number  of  permitted  subtest  failures. 


P  B 

These  equations  are  solved  numerically  for  r  and  r  in  Figures  8-5  and  8-6, 

P  B 

respectively,  for  R  =  R  =  10%.  As  an  example  with  N  =  100  and  m  =  10  for  example 

P  B  ® 

r  *  7%  and  r  *  85%,  somewhat  better  than  for  the  case  where  :o  failures  were  permitted 

Each  test  situation  will  demand  its  own  tradeoffs  and  these  ideas  are  only  to  focus  attention 

on  the  procedures  for  effectively  designing  tests  which  are  statistically  fair  to  producer  and 

buyer  alike. 
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Figure  8-C.  Producer's  Risk  per  Test  vs  Total  Number  of  Tests  with  Overall 
Risk  of  10%  and  the  Number  of  Permitted  Failures  as  a  Parameter 
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Figure  8-6  ,  Buyer's  Risk  per  Test  vs  Total  No.  of  Tests  with  Overall  Risk  of  10% 
and  the  No.  of  Permitted  Failures  as  a  Parameter 
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APPENDIX  A 
DERIVATION  OF  THE  TRACE  v 


From  Equation  (5-5), 


v  =  (I  -  M)  (I.-  M) 


where 


T  -1  T 
M  =  P  (P  P)  P 


and  P  is  partitioned  horizontally  as, 
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(N-8  x  M+l) 


22 


(A-l) 


(A-2) 


(A-3) 


resulting  in  a  two-dimensional  partition  of  M  as 
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Defining 
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(M+l  X  M+l) 


one  finds  for  orthogonal  polynomials  that  the  matrix  is  diagonal  with  elements 


i  =  Z  p2 
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without  any  partitioning.  The  structure  of  M  is  thus 
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Now 


-  <L-M>2  (L-  m)2 

(NxN)  (Nxs-r)  (s-rxN) 

and 

Trace  v ^  =  Trace  (I  -  M)  2  d~  M)g 

=  Trace  (I.-  M)2  (I*  M>  * 

=  Trace  jx^  . 

Thus  the  ij^1  partitioning  of  jx  is  thus 
3 

£  (L-M)u  (L-S>J, 
i  =1 

But  only  the  diagonal  elements  of  jx  are  needed  for  the  Trace p22 
and 

M  ji  =  (I-M)u(I-M)^  +  (I-M)i2a-M)2{,+  (i-M)i3(I-M)Ji  (A-12) 
Specifically  from  Equation  (A-8), 

^22  =  M21  M12  +  ^  M22*  +  M23  M32 

(s-r  x  s-r)  (s-r  x  r  x  r  x  s-r)  (s-r  x  s-r  x  s-r  x  s-r)  (s-r  x  N-s  x  N-s  x  s-r) 

(A-13) 
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Expansion  yields, 
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where  the  middle  term  reflects  the  different  structure  of  the  diagonal  of  I  - 
that 

s 

"nr  X 

i=r+l 


In  general  and 


-  M)j)  ♦ 


for  n  ^  j  and 


nn 


X 

i=r+l 

i^n 


(A- 14) 


M.  noting 


(A-15) 


for  n  =  j  in  particular. 


Combining  the  terms  in  Equation  (A- 14),  adding  and  subtracting 
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one  gets 
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from  successive  application  of  the  orthogonality  relations  [Equation  (4-5)] 
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Introducing  this  result  into  Equation  (A- 16),  expanding  and  cancelling  terms, 


Trace  v 


*_S2=  L 


M+l  ^  2 


n=r+l 


and  finally  with  the  definition  of  B^, 


Tr*c»  \2  =  * 


M+i  y 
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APPENDIX  B 


DISTRIBUTION  FOR  THE  PARTITIONED  DATA  SET 


From  Equations  <6 -4)  and  (5-6 A),  we  have  that 


A 

2 

a  = 


Z'TZ'  S,T^2.2' 

Trace  v  ~  Trace  v'22 


(B-l) 


so  that 


m  - 


(B-2) 


In  general,  the  estimate  of  variance  involves  the  summation  of  L=N  squares  of  partially 
correlated  zero-mean  GaussiAn  variables.  Zero-correlation  (totally  independent  variables) 
requires  that  each  variable  be  zero-mean  a  priori.  In  this  case,  the  estimate  ic  chi-square 
distributed  with  L  degrees  of  freedom.  Unity  correlation  (completely  dependent  variables) 
requires  that  each  variable  be  essentially  the  same.  In  the  case,  the  estimate  is  still  chi- 
square,  but  with  1  degree  of  freedom.  Moreover  we  have  seen  that  estimation  of  an  Mth 
order  polynomial  trend  model  uses  up  M+l  degrees  of  freedom,  partially  correlating  the 
data  in  the  process.  In  this  case,  the  estimate  is  chi-square  distributed  with  L-(M+I) 
degrees  of  freedom.  In  summary,  the  summation  of  L  squared  zero-mean,  partially 
correlated,  Gaussian  variables  always  leads  to  a  chi-square  distribution  with  v  =  l  -  x 
degrees  of  freedom.  For  the  partitioned  case,  L  =  s-r  and 


(B-3) 


Note  that  x  =  0  for  p  =  0  and  x  =  L-l  for  p  =  -1,  i.  e. ,  M  =  N-l. 
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With  this  line  of  reasoning,  we  can  use  Equation  (B-l)  to  redefine  the  variance 


estimate  as 


XS  a2  2 
2  *  v 

*  *  Trace 


(B-4) 


Thus, 


E(0) 


Trace  v 


S-rr-  E(x„2) 


Tr.ce 


(B-5) 


From  Equations  (B-2)  and  (B-5)  then 


v  *  Trace  » 


(B-€) 


the  number  of  degrees  of  freedom  for  the  general  case. 
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APPENDIX  C 


MONTE-CARLO  SIMULATIONS  TO  VALIDATE  THE  TECHNIQUE 

Many  simulations  were  performed  to  help  validate  the  technique  in  increasingly 
realistic  steps.  The  common  elements  of  each  simulation  were  as  follows: 

1.  Gaussian  noise  was  added  to  simple  models  for  the  mean  flight 
path  (trend). 

•  independent  samples 

•  representative  models  for  radar  measurement  noise  including 
range  dependent  and  range  independent  components. 

2.  The  target  was  assumed  to  have  a  constant  speed  and  mean  cross  section, 
typical  of  small  military  aircraft. 

3.  The  radar  rotated  at  5  rpm  and  was  assumed  to  have  typical  signal 
processing  features  and  radar  coverage. 

Two  simulations  will  be  reported  which  both  illustrate  the  value  and  validity 
of  specific  features  of  the  technique  developed  here  and  demonstrate  its  robustness  as 
well. 

C.  1  TRENDS  OF  KNOWN  AND  CONSTANT  POLYNOMIAL  ORDERS 

The  first  simulation  of  Interest  is  one  in  which  the  mean  flight  path  was  linear  in 
range  and  constant  in  azimuth,  i.e.  truly  radial,  so  as  to  provide  a  closely  controlled 
test.  Each  data  interval  included  L=10  points  for  the  variance  estimate  and  N^30  points 
for  the  trend  removal  process.  A  detection  probability  of  90%  was  assumed  at  all  ranges 
and  500  Monte-Carlos  were  performed.  Each  Monte-Carlo  v/as  independent,  whereby 
samples  were  drawn  first  from  Gaussian  random  number  generators  scaled  for  range  and 
azimuth  position  errors,  and  then  from  a  uniform  random  number  generator  (simulating 
a  90/10  coin  flip)  to  determine  radar  detectability.  After  a  30-point  data  set  was  assem¬ 
bled,  the  estimation  process  was  initiated  and  the  autocorrelation  and  variance  estimates 
were  stored.  This  process  was  repented  500  times  (all  independent)  and  the  estimates 
averaged  to  approximate  an  expectation  value.  Typical  results  are  shown  in  Table  C-l, 


RESULTS  OF  SIMULATION  WITH  A  KNOWN  AND  CONSTANT  TREND  MODEL 
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The  upper  set  of  tables  show  the  actual  range  and  azimuth  of  the  aircraft,  and  the  noise 
models  for  root-track  position  variance  used  to  scale  the  Gaussian  random  number  gen¬ 
erators  at  each  range  point.  Data  points  11  through  20  have  asterisks  indicating  those 
used  to  estimate  variance.  The  lower  set  of  tables  show  the  results  versus  assumed 
order  of  fit.  These  tables  are  grouped  first  for  range  measurement  and  second  for  azimuth 
measurement.  Within  each  grouping  are  two  tables,  the  first  summarizing  performance  in 
estimating  the  adjacent  point  autocorrelation  function  and  the  second  summarizing  per¬ 
formance  in  estimating  the  root-variance  of  radar  measurement  error.  Both  truth  (which 
we  know  with  certainty  in  this  case)  and  results  (labeled  "expected”)  are  given  for  compari¬ 
son.  The  following  summarizes  the  results: 


1.  In  all  cases  where  the  correlation  coefficient  is  negative,  the  root-variance 
estimates  are  excellent,  exhibiting  the  level  of  deviation  from  truth  expected 
from  the  sample  size  of  500. 

2.  The  correlation  coefficient  goes  negative  for  a  first  order  fit  in  range  and  a 
zero  order  fit  for  azimuth,  as  predicted. 

3.  A1J  estimates  with  overfitting  are  excellent  with  very  little  growth  seen  in 
the  standard  deviations  of  estimation  error,  as  indicated  by  the  "4/-" 
symbols. 


4.  The  Monte-Carlo  calculated  values  for  uncertainly  in  the  variance  estimate 
are  very  close  to  theoretical  expectation.  For  example  consider  M=6. 
From  Figure  8-2  with  N=30  and  L=10.  v  «7.  From  Equation  (7-8), 


VAR  (<r2) 


(C-l) 


From  Table  C-l  and  M=8,  <r  _  =  114.  3  ft  and  a  .  =  1.682  mrad.  Then  from 
Equation  (C-l),  K  A 


.83.6  ft 


1.  23  mrad 


VAR1/4  (a2) 


which  compare  quite  favorably  to  the  Mcnte-Carlo  results  reported,  i.e. , 
81. 1  ft  and  1. 213  respectively. 
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C.  2  COMPLEX  TRIGOMETRIC  TREND  MODELS 


An  example  is  given  here  of  a  Monte-Carlo  simulation  that  was  actually  used  as  an 
acceptance  test  fox*  the  data  reduction  technique  by  a  GE  customer.  An  aircraft  was  sim  ¬ 
ulated  to  be  flying  at  40  left  at  a  speed  of  487  knots.  It  flew  10  east- west  legs  out  to  200  nmi 

with  a  north  offset  of  0. 5  nmi.  The  actual  aircraft  flight  path  oscillated  about  the  radial  in 

both  plan-position  and  heigh:  according  to  the  following  equations: 

N  =  0. 5  nmi  +  0. 0234  SIN  (42. TEt^)  (C-2) 

H  =  40, 000  ft  +  300  SIN  (43. 2  E  +$,.)  (C-3) 

where 

E  =  distance  east  of  the  radar  (nmi) 

N  =  distance  north  of  the  radar  (nmi) 

<t>  =  arbitrary  phase  angles  (deg),  and 
H  =  altitude  (ft)  . 

At  the  start  of  each  leg,  the  phase  was  reinitialized  independently  for  distance  and 
altitude.  The  maximum  aircraft  amplitude  deviation  from  radial  was  142  ft  with  a  period 
of  €2. 3  s  (8.43  nmi).  The  maximum  amplitude  deviation  in  height  was  300  ft  with  ? 
period  of  61. 57  s  (8. 33  nmi). 

On  top  of  this  mean  aircraft  motion  was  measurement  noise  orawD  independently 
point-to-point  from  a  Gaussian  random  number  generator  f  caled  to  a  typical  model 
for  noise  vs  range. 

Pass/luil  thresholds  were  calculated  based  on  a  1%  risk  of  talsoly  failing  due  to 
statistical  uncertainty.  Sixty  tests  were  made  in  all  (2C  range  intervals  each  for  range, 
height,  and  azimuth).  The  results  are  given  in  Tables  C-2,  C-3  and  C-4  for  range, 
height,  and  azimuth,  respectively.  The  first  two  columns  give  the  range  intervals.  The 
second  column  shows  the  theoretical  standard  deviation  of  measurement  erro ;  from  which 
the  noise  was  added  to  the  data,  The  fourth  column  gives  the  estimates  made  by  the  tech¬ 
nique  described  here  and  the  last  column  presents  the  pass/fail  thre&r-olds. 
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TABLE  C-3 

SIMULATION  RESULTS  FOR 
HEIGHT  ACCURACY 


IEORY 

mi 


PASS/FAIL 

(FEET) 


ESTIMATION 
-(FEET) 


TABLE  C-4 

SIMULATION  RESULTS  FOR 
AZIMUTH  ACCURACY 


miu 

AMU 

mm 

AIM 


iM 

L’l#’ 


The  a  priori  expectation  of  falsely  failing  one  of  the  60  tests  is  the  risk  times  the 
number  of  tests,  i.  e. ,  1%  x  60  =  60%.  It  is  interesting  that  no  failures  occured.  This  is 
indeed  remarkable  particularly  for  height  accuracy  close  to  the  radar  where  the  amplitude 
of  the  real  aircraft  motion  was  almost  three  times  larger  than  the  rms  noise  level. 


APPENDIX  D 


SAMPLE  RESULTS  FOR  A  REAL  FLIGHT  TEST 

In  Appendix  C  we  saw  how  well  the  technique  worked  for  simulated  flight  trends 
with  known  and  constant  polynomial  orders  on  the  one  hand  and  complex  trigonometric 
paths  of  variable  and  unknown  order  on  the  other.  In  the  first  case,  500  Monte-Carlos 
were  performed  so  that  the  estimation  error  was  small.  In  the  second  example,  10  flight 
legs  were  averaged,  equivalent  to  10  Monte-Carlos,  and  the  larger  fluctuations  were  ac  ¬ 
commodated  by  pass/fail  thresholds  calculated  from  the  risk  concepts  of  Section  VIII.  Here 
we  show  results  for  a  single  range  interval  and  a  single  leg  (1-Monte-Carlo)  from  an  actual 
flight  test  using  a  Lear  jet  on  a  near-radial  flight  path. 

Figure  D-l  shows  one  leg  that  we  will  examine  in  detail.  Tables  D-l,  D-2,  and  D-3 
summarise  the  results  for  the  interval  between  30  and  40  nmi  for  range,  azimuth,  and  height 
errors  respectively,  versus  assumed  order  of  fit.  Because  we  have  in  effect  only  one  Monte- 
Carlo  these  are  instantaneous  estimates,  not  expectation  values.  It  is  not  surprising  then  to 
see  reasonable  estimates  corresponding  to  positive  values  of  correlation  coefficient.  Never¬ 
theless  the  following  conclusions  may  be  drawn  from  this  example: 

1.  For  all  three  coordinates,  an  asymptotic  estimate  is  reached  with  remarkably 
low  fluctuation. 

2.  The  coordinates  start  off  with  truly  poor  estimates  when  the  ins  tantaneous 
correlation  coefficient  is  above  0. 8. 

3.  Even  though  the  pilot  was  attempting  to  fly  a  constant  height,  radial  flight 

path  (i.e. ,  Kp  =  1,  =  0,  =  0)  the  estimates  for  these  implied  orders 

of  fit  are  truly  poor. 

4.  Estimates  of  the  theoretical  values  are: 

<xR  =  114  ft 

a  .  =  0. 79  mrad 

A 

a  H  =  229  ft 
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Note  that  by  applying  the  rule  for  picking  the  best  estimate  at  ranges  less 
than  40  nmi  (see  par.  7. 1),  estimates  are  chosen  with  the  fewest  points, 
N,  and  the  highest  order  of  fit  such  that  negative  autocorrelations  are 
achieved  for  M-l,  M  and  M+l.  From  Tables  D-l,  D-2,  and  D-3  then  the 
selected  estimates  are: 

oR  =  150  ft 

o  A  =  0. 86  mrad 

o  H  =  205  ft 

which  are  not  bad  for  instantaneous  estimates. 


TABLE  D-l 


RANGE  ERROR  ESTIMATION  FOR  30-40  nmi  RANGE  INTERVAL 

N  =L=8 


M 

CORR 

SIGMA  (FEET) 

TRACE 

0 

0.883 

19583.7 

7.00 

1 

-0.426 

117.6 

6.00 

2 

-0.502 

122.2 

5.00 

3 

-0.459 

131.7 

4.00 

4 

-0.548 

149.5* 

3.00 

5 

-0.645 

159.9 

2.00 

♦This  is  the  selected  estimate. 


TABLE  D-2 

AZIMUTH  ERROR  ESTIMATION  FOR  30-40  nmi  RANGE  INTERVAL 

N=L=8 


M 

CORR 

SIGMA  (MRAD) 

TRACE 

0 

0.834 

3.900 

7.00 

1 

-0.663 

0.693 

6.00 

2 

-0.729 

0.755 

5.00 

3 

-0.733 

0.836 

4.00 

4 

-0.677 

0.857* 

3.00 

5 

-0.726 

1.030 

2.00 

♦This  is 

i  the  selected  estimate. 
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TABLE  D-3 

HEIGHT  ERROR  ESTIMATION  FOR  30-40  nmi  RANGE  INTERVAL 

N=L^8 


M 

CORR 

SIGMA  (FEET) 

TRACE 

0 

0.162 

231.5 

7.00 

1 

-0.114 

194.2 

6.00 

2 

-0.058 

206.3 

5.00 

3 

-0.392 

204.0 

4.00 

4 

-0.393 

204.8* 

3.00 

5 

-0.629 

65.6 

2.00 

"“This  is  the  selected  estimate. 

Figure  D-2  shows  all  of  the  legs  combined  for  thiB  flight  test.  Unfortunately  for 
die  (30-40  nmi)  range  interval  there  were  only  2  legs.  The  estimates  are: 

aR  =  115  ft 

=  0. 862  mrad 

<rH  -  236  ft 

in  some  cases  closer  to  our  theoretical  expectations.  To  demonstrate  that  these  estimates 
are  truly  consistent  with  theory  within  the  limits  of  sampling-induced  uncertainty,  consider 
that  the  sum  of  the  2  traces  from  Tables  D-l,  D-2,  and  D-3  are  14. 3,  14. 8,  and  15. 7  re¬ 
spectively.  The  square-root  of  the  upper  and  lower  90%  confidence  bounds  to  the  above  esti¬ 
mates  are  shown  below  as  bracketing  theory  in  all  these  cases. 

RANGE 

88.4  <  114ft  <  168 

AZIMUTH 

0.663  <  0.79  mrad  <  1.26 

HEIGHT 

183  <  229  ft  <  339  . 


Figure  D-2.  PPI  Plot  (All  Legs  Combined) 


As  an  example  of  the  method  used  beyond  40  nm,  based  on  the  leg  plotted  In  Figure 
D-l,  Tables  D-4,  D-5,  and  D-6  summarize  the  results  for  the  interval  between  50  and  60  nmi 
for  range,  azimuth,  and  height  errors  respectively  versus  the  best  estimate  for  each  number 
of  points,  N,  from  L  to  4L. 

The  estimates  of  the  theoretical  values  for  this  case  are: 
aR  =  114  ft 

<rA  =  0.4  mrad 

<rH  *  310  ft 

This  time  the  rule  for  picking  the  best  estimate  at  ranges  greater  than  40  nmi  is 
applied  (see  par.  7. 1).  For  each  number  of  points,  candidates  are  chosen  with  the  minimum 
order  of  fit,  M,  such  that  negative  autocorrelations  are  achieved  for  M-2,  M-l  and  M.  The 
candidate  with  the  maximum  trace  is  then  chosen  as  the  best  estimate.  For  Tables  D-4, 

D-5,  and  D-6,  the  best  estimates  are: 

oR  =  121  ft 

a  A  =  0.45  mrad 

ff  H  =  241  ft 

Again,  from  Figure  D-2,  only  two  legs  (p=2)  were  available  for  this  range  interval 
(30-60  nmi)  with  the  2-leg  average  given  as: 

SR  =  93.7  ft 

a.  -  0.411  mrad 

A 

A 

ffH 


237.6  ft 
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TABLE  D-4 

RANGE  ERROR  ESTIMATION  FOR  50-60  nml  RANGE  INTERVAL 


>• 

CORR 

litai 

CORR 

iHtil 

CORR 

l&ll 

CORR 

JILL 

M 

SIGMA  (FEET) 

TRACE 

8 

0,382 

-0.250 

-C.250 

-0.513 

3 

140.6 

4.00 

9 

0.909 

-0.241 

-G.246 

-0.263 

3 

136.6 

4.86 

10 

0.927 

-0.235 

-0.235 

-0.192 

3 

130.3 

5.65 

11 

0.941 

-0.331 

-0.292 

-0.304 

3 

125.6 

5.87 

12 

0.951 

-0.205 

-0.244 

-0.242 

3 

126.5 

6.07 

13 

0.959 

-0.095 

-0.295 

-0.281 

3 

124.1 

6.24 

14 

0.103 

-0.255 

-0.258 

-0.334 

4 

124.3 

6.01 

1b 

0.164 

-0.283 

-0.278 

-0.376 

4 

122.5 

6.17 

ie 

0.324 

-0.267- 

-0.267 

-0.355 

4 

122.5 

6.33 

17 

0.430 

-0.323 

-0.319 

-0.370 

4 

122.0 

6.45 

le 

0.540 

-0.316 

-0.321 

-0.350 

4 

122.1 

6.55 

19 

0.554 

-0.157 

-0.230 

-0.214 

4 

123.3 

6.62 

29 

0.637 

-0.150 

-0.222 

-0.212 

4 

122.8 

6.70 

21 

0.617 

-0.144 

-0.283 

-0.290 

4 

120.9 

6.76 

22 

0.600 

-0.162 

-0.145 

-0.228 

4 

128.5 

6.82 

23 

0.635 

-0.158 

-0.173 

-0.216 

4 

128.0 

6.87 

24 

0.580 

-0.263 

-0.253 

-0.369 

4 

121.1 

6.92 

25 

0.599 

-0.277 

-0.283 

-0.340 

4 

121,3 

6.96 

26 

0.656 

-0.303 

-0.303 

-0.364 

4 

121.9 

7.00 

27 

0.693 

-0.312 

-0.325 

-0.373 

4 

121.4 

7.U3 

r\ 

\0.722 

-0.293 

-0.293 

-0.374 

4 

121.0 

7.06 

0.739 

-0.309 

-0.308 

-0.371 

4 

122.0 

7.09 

30 

0.764 

-0.293 

-0.290 

-0.370 

4 

121.7 

7.12 

31 

0.733 

-0.245 

-0.250 

-0.261 

4 

124,2 

7.15 

32 

0.826 

-0.261 

-0.c34 

-0.289 

4 

123.0 

7.17 

33 

0.839 

-0.274 

-o.2r,3 

-0.289 

4 

121.3 

7.20* 

*  This  is  the  selected  estimate. 
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TABLE  D-5 

AZIMUTH  ERROR  ESTIMATION  FOR  50-60  nmi  RANGE  INTERVAL 


777 

•< 

852 

-i 

879 

>i 

CORR 

iibii 

-0,400 

■0.440 

-0.369 

•0.395 

■0.3;7 

•0.196 

■0.153 

-0.261 

-0.258 

-0.080 

-0.061 

-0.102 


CORR 


SIGMA  (mrad 


3 

0.541 

4.00 

3 

0.503 

4.86 

3 

0.476 

5.65 

3 

0.463 

5.87 

3 

0.451 

6.07 

3 

0.446 

6.24 

3 

0.439 

6.37 

3 

0.464 

6.49 

3 

0.461 

6.60 

3 

0.477 

6.70 

3 

0.484 

6.78 

3 

0.463 

6.86 

3 

0.457 

6.93 

4 

0.458 

6.76 

4 

0.432 

6.82 

7 

0.472 

6.14 

5 

0.467 

6.66 

5 

0.445 

6.72 

5 

0.443 

6.77 

5 

0.455 

6.82 

5 

0.452 

6.87 

6 

0.449 

6.76 

6 

0.430 

6.80 

5 

0.440 

7.00 

5 

0.445 

7.03 

6 

0.433 

6.91 

This  is  the  selected  est  mate. 


TABLE  D-6 


HEIGHT  EBROR  ESTIMATION  FOR  50-60  nmi  RANGE  INTERVAL 


N 

CORR 

IM=3I 

CORR 

iml 

CORR 
{M-l ) 

CORR 

JMi 

M 

SIGMA  (FEET) 

TRACE 

8 

0.051 

-0.329 

-0.396 

-0.667 

5 

166.2 

2.00 

9 

0.075 

-0.160 

-0.207 

-0.609 

5 

209.7 

3.00 

10 

0.085 

-0.067 

-0.084 

-0.441 

5 

217.5 

3.97 

11 

0.089 

-0.007 

-0.020 

-0.282 

5 

220.3 

4.59 

12 

0.026 

-0.185 

-0.211 

-0.520 

7 

191.1 

3.85 

13 

0.051 

-0.106 

-0.129 

-0.378 

7 

199.4 

4.30 

14 

0.044 

-0.038 

-0.054 

-0.289 

7 

207.6 

4.64 

15 

0.006 

-0.002 

-0.202 

-0.233 

8 

221.2 

4.50 

16 

0.001 

-0.043 

-0.013 

-0.126 

7 

234.6 

5.22 

17 

0.128 

-0.007 

-0.000 

-0.007 

5 

237.9 

6.07 

18 

0.007 

-0.047 

-0.050 

-0.048 

6 

245.5 

5.87 

19 

0.054 

-0.057 

-0.050 

-0.097 

5 

233.8 

6.28 

20 

0.019 

-0.048 

-0.085 

-0.098 

5 

232.2 

6.37 

21 

0.007 

-0.052 

-0.050 

-0.149 

5 

231.6 

6.45 

22 

0.604 

-0.015 

-0.058 

-0.126 

3 

228.5 

7.04 

23 

0.636 

-0.047 

-0.078 

-0.122 

3 

230.8 

7.09 

24 

0.547 

-0.181 

-0.241 

-0.216 

3 

237.6 

7.14 

25 

0.576 

-0.196 

-0.237 

-0.225 

3 

236.6 

7.18 

26 

0.542 

-0.161 

-0.274 

-0.273 

3 

233.5 

7.22 

27 

0.606 

-0.108 

-0.269 

-0.264 

3 

231.9 

7.25 

28 

0.539 

-0.074 

-0.283 

-0.284 

3 

241.5 

7.28 

29 

0.613 

-0.003 

-0.283 

-0.271 

3 

240.9 

7.31* 

30 

0.106 

-0.285 

-0.304 

-0.383 

4 

236.3 

7.12 

31 

0.106 

-0.273 

-0.310 

-0.335 

4 

240.0 

7.15 

32 

0.106 

-0.191 

-0.278 

-0.276 

4 

242.3 

7.17 

33 

0.106 

-0.180 

-0.219 

-0.210 

4 

240.1 

7.20 

♦This  is  the  selected  estimate. 
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To  demonstrate  that  these  estimates  are  truly  consistent  with  the  theory  within  the  limits  of 
sampling-induced  uncertainty,  consider  that  the  sum  of  the  two  traces  are  13.6,  14.  2,  and 
14. 3  for  range,  azimuth,  and  height  respectively.  Once  again  the  square-root  of  the  upper 
and  lower  90%  confidence  bounds  for  the  estimates  are  shown  below  as  bracketing  theory  in 
all  these  cases. 


i 

i 

f 

\ 

t 


RANGE  I 

« 

71.4  <  114  ft  <  139  j 

AZIMUTH  j 

0.316  <  0.4  mrad  <  0.600 

HEIGHT 

\ 

183  <  310  ft  <  347 

Figures  D-3  through  D-7  show  the  results  for  all  legs  included  in  this  flight  test 
(plotted  in  Figure  D-2).  Figure  D-3  shows  the  probability  of  detection  based  on  10-nmi 
range  intervals.  Not  until  the  size  of  the  range  intervals  is  reduced  (Figure  D-4)  is  multi-  • 

path  interference  observable.  Figures  D-5,  D-6,  and  D-7  plot  the  averaged  estimates  for  | 

range,  azimuth  and  height  error  respectively.  These  are  the  standard  plotted  outputs  avail-  j 

able  from  the  statistics  package  constructed  from  the  concepts  introduced  in  this  report.  1 

* 

| 

« 
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Figure  D-7.  Standard  Deviation  Plot  Demonstration  (Height) 


APPENDIX  E 


STATISTICAL  COMPARISON  OF  RESULTS  FROM  A  REAL-FLIGHT  TEST 
WITH  A  CO-LOCATED  PRECISION  REFERENCE  STANDARD 


This  Appendix  documents  a  recent  opportunity  to  validate  the  autonomous  technique 
by  direct  comparison  with  the  results  obtained  by  tha  more  conventional  method,  i.  e. ,  using 
a  precision  reference  standard. 

Comparisons  were  made  for  approximately  6  flight  legs  for  ranges  between  20  and 
200  nmi.  A  small  aircraft  flew  closely  controlled  radial  legs  and  the  results  for  both  tech¬ 
niques  are  presented  for  20  nmi  range  bins  in  Figures  E-l,  E-2,  and  E-3  for  the  standard 
deviation  of  range,  azimuth,  and  height  respectively.  The  dark  solid  lines  are  the  results 
for  the  autonomous  method,  while  the  lighter  solid  lines  are  the  results  from  comparison 
with  a  precision  reference  standard.  Both  estimates  are  independent  random  variables 
because  of  the  limited  sample  size  <7  to  14  samples  per  range  bin).  Because  these  variance 
estimates  are  subject  to  sampling  uncertainty,  I  have  also  shown  95%  confidence  bounds  for 
both  methods.  The  dotted  lines  are  bounds  for  the  autonomous  method,  while  the  dashed 
lines  are  bounds  for  the  method  using  the  reference  standard.  The  cross-hatched  areas 
represent  regions  of  overlap.  A  reasonable  interpretation  is  that  if  either  estimate  lies 
within  an  overlapped  region,  then  the  two  estimates  can  be  considered  consistent  with  95% 
confidence.  Estimates  for  all  three  coordinates  are  consistent  between  the  two  methods  for 
all  nine  range  bins. 
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Figure  E-2.  Statistical  Comparison  of  Direct  Method  to 
Autonomous  Technique  for  Estimating 
1  Sigma  Azimuth  Error 
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Figure  E- 


3.  Statistical  Comparison  of  Direct  Method  to 
Autonomous  Technique  for  Estimating 
1  Sigma  Height  Error 


APPENDIX  F 


AN  EXAMPLE  OF  THE  INVERSE  CHARACTERISTIC  FUNCTION 
BY  DISCRETE  FOURIER  TRANSFORMS 

From  par.  8. 1,  it  was  stated  that  inverse  characteristic  functions  could  be  numeri¬ 
cally  computed  from  discrete  Fourier  transforms.  Consider  the  Gamma  distribution  as  an 
example, 


g(t)  « 


where 


0-1  e-t/fl 


a  =  v/2. 


and 


0  = 


and  which  has  a  characteristic  function 


(P-1) 


(P-2) 


(F-3) 


*<“>  *  (- aTJZ ■)  •  ,F-4) 

Suppose  we  have  a  random  variable  which  is  the  sum  of  p  gamma-distributed  variates. 
Thee  the  characteristic  function  of  this  sum  is 


a 


i 


which  has  no  known  general  inverse  for  arbitrary  j  v. 


(F-5) 


>^v* 


F-l 


■A  V.  . 


The  distribution  for  the  general  sum  can  be  obtained  from  the  inverse  characteristic 

function 


S  (t)  = 
£ 


r  ir  Kk) '*• 

-«  icp  '  1  7 


e‘Jwt  d«. 


(F-6) 


As  a  discrete  Fourier  transform, 


a  . 
g  (k) 
£ 


.  /  V2  \"l/2 

=  ?  7^  (  V2-J"n/ 


-i“nk 


(F-7) 


n=0  iep 


k  =  0,  1,  2,- 


1  N-l, 


2  2 

where  t  is  a  sample  of  the  random  variable  a  / a  and  we  evenly  sample  all  of  t-space 

£ 

at  least  2N  times.  Note  that  when  all  |  are  equal  for  each  leg  that 


0  -  \ ;  a  -  p  p/2  (F-8) 

so  that  the  distribution  becomes  gamma  again  with  p  Trace  v'  degrees  of  freedom. 

Figure  F-l  shows  the  results  of  such  a  numerical  inverse  for  N=4096  with  ten  legs, 
each  having  v  =6,  so  that  =  1/3  and  0  =  30.  Here  we  have  plotted  the  cumulative  distri¬ 
bution,  i.  e.  solid  line,  since  this  is  the  statistic  important  to  risk  and  confidence  consider¬ 
ations.  The  circles  are  samples  of  the  theoretical  gamma  distribution  with  the  same 
parameters.  The  agreement  is  excellent  over  the  entire  range  of  interest.  To  illustrate 

the  error  encountered  by  assuming  that  the  sum  £  v,  is  the  only  parameter  of  importance, 

iep  1 

consider  the  second  curve  (dashed)  in  Figure  F-l  which  preserves  i  re  total  number  of  de¬ 
grees  of  freedom,  i.  e.  60,  but  which  has  different  values  of  from  leg-to-leg.  The  differ¬ 
ence  can  be  quite  large.  For  example,  if  a  producer's  risk  of  0. 1%  is  desired,  then  the 
true  pass/fail  threshold  would  be  ^18. 7  a  instead  of  the  assumed  threshold  of  ^16. 5  a  , 
a  difference  of  about  6. 5%. 
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mm 

s 

0. 1  cm 

lb 

b 

cm 

= 

0. 3937  in. 

lb 

c 

cm 

B 

0.  032e  ft 

metric  ton 

B 

cm 

SS 

10  mm 

cm2 

9 

m 

B 

= 

0.1550  in. 

cm2 

B 

1.076  •  10’3  ft2 
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Q 

m 

B 

cm 

S 
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2 

cm3 

S 

3.531  •  10‘5  ft3 

m 

2 

B 

m 

= 

ft 

s 

30.48  cm 

m3 

S 

ft 

s 

0.3048  m 

m3 

ft2 

s 

0. 0929  m2 

*2 

o 

mi 

B 

ft 

s 

929.37  cnri 

Mm 

(micron) 


9.294  •  10"3  km2 
0. 0283  m3 

2. 54  cm 
6.452  cm2 
16.387  cm2 

0. 001  mm 


mi 

mi/h 


453. 6  g 
0. 4536  kft 

1. 12  tons  (U.S. ) 

39.37  in. 
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1.0936  yd 
10. 76  ft2 
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5280  ft 
0. 87  nmi 
1760  yd 
2.59  km2 
0. 87  knots 

1.852  km 
6076  ft 


Mm 

10  6m 

nmi 

* 

1. 15  mi 

Mm 

10'4  cm 

yd 

B 

0. 9144  m 

Min. 

B 

2.54  ■  10  5mm 

yd2 

= 

0.836  m2 
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m 

2.2046  lbs 

yd3 

se 

0.  7645  m3 

km 

B 
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qt 

B 
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km 

S 
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liter 

S 

1.057  qt 

km 

B 
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B 
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B 

1. 076  •  107  ft2 
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«= 

4046.  72  m2 

km2 

B 

0.381  ml2 
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E 

57.2958* 
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S 
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* 

0,  017  rad 
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K 

3.152  mi/h 
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GENERAL  ELECTRIC  COMPANY  TECHNICAL  INFORMATION 


Within  the  limitations  imposed  by  Government  data  export  regulations  and  security 
classif ications ,  the  availability  of  General  Electric  Company  technical  information 
is  regulated  by  tho  following  classifications  in  order  to  safeguard  proprietary  infor¬ 
mation: 

CLASS  1:  GENERAL  INFORMATION 

Available  to  anyone  on  request  ■ 

Patent,  legal  and  commercial  review 
required  before  issue. 

CLASS  2:  GENERAL.  COMPANY  INFORMATION 

Available  to  any  General  Electric  Company 
employee  on  request. 

Available  to  any  General  Electric  Subsidiary 
or  Licensee  subject  to  existing  agreements. 

Disclosure  outside  General  Electric  Company 
requires  approval  of  originating  component. 

CLASS  3:  LIMITED  AVAILABILITY  INFORMATION 

Original  Distribution  to  those  Individuals  with 
specific  need  for  information. 

Subsequent  Company  availability  requires 
originating  component  approval . 

Disclosure  outside  Gene. . 1  Electric  Company 
requires  approval  of  originating  component . 

CLASS  4:  HIGHLY  RESTRICTED  DISTRIBUTION 

Original  distribution  to  those  Individuals  person¬ 
ally  responsible  for  the  Company's  Interests  in 
the  subject. 

Copies  serially  numbered,  assigned  and  recorded 
by  name. 

Material  content,  and  knowledge  of  existence, 
restricted  to  copy  holder . 

GOVERNMENT  SECURITY  CLASSIFICATIONS,  when  required,  take  precedence  > 

in  the  handling  of  the  material.  Wherever  not  specifically  disallowed,  the  General 
Electric  classifications  should  also  be  included  In  order  to  obtain  proper  handling 
routines. 
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